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Through the ERIC Clearinghouse on Assessment and Evaluation, we are making the 
software and manual available free on the internet for non-commercial, educational and 
research purposes. Should you prepare a report or paper based on the use of this 
software, we require that you acknowledge Meta-Stat in your report or paper. 



WE SPECIFICALLY DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, 
INCLUDING BUT NOT LIMITED TO, IMPLIED WARRANTIES OF 
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE WITH 
RESPECT TO DEFECTS IN THE DISKETTE AND DOCUMENTATION, AND THE 
PROGRAM LICENSE GRANTED HEREIN, IN PARTICULAR, AND WITHOUT 
LIMITING OPERATION OF THE PROGRAM LICENSE WITH RESPECT TO ANY 
PARTICULAR APPLICATION USE, OR PURPOSE. IN NO EVENT SHALL WE BE 
LIABLE FOR ANY LOSS OF PROFIT OR ANY OTHER COMMERCIAL DAMAGE, 
INCLUDING BUT NOT LIMITED TO SPECIAL, INCIDENTAL, CONSEQUENTIAL 
OR OTHER DAMAGES. 

Meta-Stat was written in Turbo Pascal 7.0 by Borland International and it incorporates 
the following toolboxes: Scientific and Engineering Tools by Quinn-Curtis; Turbo 
Toolkit by Technojock Software; BGI Driver Toolkit by Ryle Design; and PCX Toolkit 
by Genus Microprogramming. 
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Preface 



Welcome to this introduction to meta-analysis 

Meta-analysis refers to the analysis of analyses... the statistical analysis of a large 
collection of analysis results from individual studies for the purpose of integrating the 
findings (Glass, 1976, p. 3). 

This book and accompanying software are intended to help you learn more about meta- 
analysis by providing you with a step-by-step guide to conducting a meta-analytic study, 
references for further research and reading, and free, high-quality software. 

Gene Glass first used the term "meta-analysis" in 1976 in his presidential address to the 
American Educational Research Association to refer to a philosophy, not a statistical 
technique. Glass argued that literature review should be as systematic as primary 
research and should interpret the results of individual studies in the context of 
distributions of findings, partially determined by study characteristics and partially 
random. Since that time, meta-analysis has become a widely accepted research tool, 
encompassing a family of procedures used in a variety of disciplines. 

Meta-analysis responds to several problems in educational research. First, important 
issues are studied by numerous investigators. The amount of information on a given topic 
therefore is often overwhelming and not amenable to summary. Even when there are 
relatively few studies on a given topic, it is difficult to determine if outcome differences 
are attributable to chance, to methodological inadequacies, or to systematic differences 
in study characteristics. Informal methods of narrative review permit biases to remain 
easily undetected. Reviewers' biases can influence decisions about study inclusion, 
relative weights given to different findings, and analysis of relations between study 
features and outcomes. These biases can have clandestine effects when reviewers do not 
systematically seek to reduce them or provide sufficient information for readers to 
evaluate their extent. 

Meta-analysis typically follows the same steps as primary research. The meta-analyst 
first defines the review's purpose. Organizing frameworks can be practical or theoretical 
questions of varying scope, but they must be clear enough to guide study selection and 
data collection. Second, sample selection consists of applying specified procedures for 
locating studies that meet specified criteria for inclusion. Typically, meta-analyses are 
comprehensive reviews of the full population of relevant studies. Third, data are 
collected from studies in two ways. Study features are coded according to the objectives 
of the review and as checks on threats to validity. Study outcomes are transformed to a 
common metric so that they can be compared. A typical metric in educational research is 
the effect size, the standardized difference between treatment and control group means. 
Finally, statistical procedures are used to investigate relations among study 
characteristics and findings. 
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The accompanying software, Meta-Stat , is a comprehensive package designed to help in 
the meta-analysis of research studies in the social and behavioral sciences. Meta-analysis 
in the traditions of Glass, Hedges and Olkin, and Schmidt and Hunter are offered. 
Features include: 

• an effect-size calculator to help convert from reported metrics to effect-size 

• Corrected regression outputs 

• Homogeneity statistics 

• Variance ratio 

• Fail-safe N. 

The system provides you with routines for easy data entry, statistical analysis, and 
graphic analysis. The data can easily be output in ASCII format or in a format ready for 
use by the Statistical Package for the Social Sciences (SPSS). 

Throughout this manual we discuss effect-size meta-analysis. Yet, Meta-Stat also offers 
correlational meta-analysis. With our easy graphic routines, you may want to use our 
Plain-Statistics module for analyzing regular, non-meta-analytic data. 

Meta-Stat has been under development for a number of years. A prototype was 
developed in 1 989 with Phase I Small Business Innovative Research (SBIR) funding 
from the National Institutes of Health. We were able to demonstrate that the system had 
potential and two years latter we were able to secure Phase II SBIR funding. In late 1993, 
the first commercial system was ready. 

Programming for Meta-Stat was conducted by Lawrence M. Rudner, David L. Evartt, 
and Patrick J. Emery. This manual was prepared by Lawrence Rudner and Dana Sohr of 
Technical Communications, Inc. in Fulton, Maryland with the assistance of Gerald 
Bracey, Gene Glass, Caroline Bagin, David Evartt, and Pamela Getson. 

The system has been through two field trials. Our field testers caught numerous mistakes, 
incorrect formulas, bugs, and the like. We gratefully acknowledge David Gibson , 
Science Applications Inc; Robert B anger t- Dr owns, State University at Albany; Philip C. 
Abrami Concordia University; Michael J. Strube , Washington University; James A. 

Kulik, University of Michigan; Chen-Lin C. Kulik , University of Michigan; Roger E. 
Millsap, City University of New York; Michael R. Stevensen Ball State University; 
Herbert C. Rudman , Michigan State University; Peter A. Cohen, Medical College of 
Georgia; Norman Miller , University of Southern California; Karl White, Utah State 
University; Larry Leslie, University of Arizona; and Arthur L. White, Ohio State 
University for their assistance with one or both of the field tests. 

We are especially indebted to Professor Gene V Glass. Gene has been a consultant to 
this project since its inception. Aside from moral support, he provided a large 
investment of time. He provided a comprehensive testing of the system catching errors 
such as the use of 1 .96 rather than the appropriate value of the t-statistic, suggested 
different ways to present information on the screens, suggested new features, helped edit 
the manual (he was very picky and he caught numerous errors) and most importantly 
reacted to ideas and answered questions. We were often in daily communication. 

I hope you find this book and Meta-Stat useful in your work and as a teaching and 
learning tool. 



Lawrence M. Rudner 
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Introducing Meta- Analysis 
and Meta-Stat 



This chapter introduces meta-analysis and the Meta-Stat program. 



Introducing Meta -Ana lysis 

Research literature, it is often pointed out, is growing at an exponential rate. A 
recent study estimated that there are 40,000 journals for the sciences, and that 
researchers are filling those journals at the rate of one article every 30 seconds, 
24 hours a day, seven days a week (Mahoney, 1985). No matter what the 
topic — from computer-aided instruction to sex differences to the effects of 
medication on hyperactivity — researchers can, in just a few years, add dozens 
and even hundreds of studies to the literature. 

As research results accumulate, it becomes increasingly difficult to understand 
what they tell us. It becomes increasingly difficult to find the knowledge in this 
flood of information. 

In 1976, Gene Glass proposed a method to integrate and summarize the findings 
from a body of research. He called the method meta-analysis. Meta-analysis is 
the statistical analysis of a collection of individual studies. 

Meta-analysis refers to the analysis of analyses. I use it to refer to the 
statistical analysis of a large collection of results from individual studies for 
the purpose of integrating the findings . It connotes a rigorous alternative to 
the casual, narrative discussions of research studies which typify our 
attempts to make sense of the rapidly expanding research literature . (Glass, 
1976) 
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The Traditional Approach to Research Integration 

Using the traditional method of integrating research studies, a reviewer provides 
a narrative, chronological discourse on previous findings. Yet this method is 
flawed and inexact: 

• Unable to deal with the large number of studies on a topic, reviewers 
focus on a small subset of studies, often without describing how the 
subset was selected. 

• Reviewers often cite the conclusions of previous reviews without 
examining those reviews critically. 

• Reviewers are usually active and prominent in the field under review. 
Therefore, they might not be inclined to give full weight to evidence that 
is contrary to their own positions. 



The Meta-Analysis Approach to Research Integration 

In a meta-analysis, research studies are collected, coded, and interpreted using 
statistical methods similar to those used in primary data analysis. The result is an 
integrated review of findings that is more objective and exact than a narrative 
review. 



Solving the Complexity Problem 

The human mind is not equipped to consider simultaneously a large number of 
alternatives. (This is true even for bright, energetic researchers.) Confronted with 
the results of 20 similar studies, the mind copes only with great difficulty. 
Confronted with 200, the mind reels. Yet that is exactly the scope of the problem 
faced by a researcher attempting to integrate the results from a large number of 
studies. As a result. 

The typical review concludes that the research is in horrible shape ; 
sometimes one gets results , sometimes one doesn 't. Then the call is sounded 
for better research designs, better measures, better statistical methods — in 
short, a plaintive wish that things were not so complicated as they are. 
(Glass, 1976) 

When performed on a computer, meta-analysis helps the reviewer surmount the 
complexity problem. The reviewer can code hundreds of studies into a data set. 
The data set can then be manipulated, measured, and displayed by the computer 
in a variety of ways. 



2 



15 



Chapter 1 




Meta-Analysis User Manual 



Introducing Meta-Analysis and Meta-Stat 



Solving the Ambiguity Problem 

Researchers can tolerate ambiguity well. Policy makers, however, particularly 
elected policy makers, have a limited time in which to act. They look to research 
to provide information that will help them choose among policy options. 

Unfortunately, original research, and narrative reviews of the research, often do 
not provide clear options to policy makers. Senator Walter Mondale expressed 
his frustration to the American Psychological Association in 1970: 

What I have not learned is what we should do about these [educational] 
problems . . . . For every study, statistical or theoretical , that contains a 
proposed solution or recommendation, there is always another equally 
well-documented study , challenging the assumptions or conclusions of 
the first. No one seems to agree with anyone else’s approach. But more 
distressing: no one seems to know what works. 



Solving the Replication Problem 

A scientific study should be designed and reported in such a way that it can be 
replicated by other researchers. However, researchers seldom attempt to replicate 
previous findings. Instead, they pursue funding for the new, the novel, or — at the 
very least — they attempt to extend what is considered to be the current state of 
knowledge in their field. The result can be an overwhelming number of studies 
on a given topic, with no two studies exactly alike. 

In such circumstances, it is difficult to determine if the differences between the 
study outcomes are due to chance, to inadequate study methods, or to systematic 
differences in the characteristics of the studies. 

Meta-analysis can help you investigate the relationship between study features 
and study outcomes. You code the study features according to the objectives of 
the review. You transform the study outcomes to a common metric so that you 
can compare the outcomes. Last, you use statistical methods to show the 
relationships between study features and outcomes. 
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Introducing Meta-Stat 

Meta-Stat is a computer program that automates the many complex tasks that are 
required to perform a meta-analysis. With Meta-Stat , you can: 

• Code the features of each study 

• Transform study findings into a common metric, called the Effect Size, so 
that the studies can be compared 

• Apply statistical techniques to the data 

• Display the data on a variety of charts 

The following sections provide an overview of Meta-Stat . 



Coding Study Features 

One of your first tasks will be to create variables with which you can code study 
features. Meta-Stat offers 10 variables as a starting point, including: 

• Effect Size 



• Unbiased Effect Size 

• Sample size 
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You can create your own variables to identify features of the studies. For 
example, suppose you are reviewing studies that investigate the effects of pre- 
test coaching on Scholastic Aptitude Test scores. You would probably need to 
create a variable to identify the number of hours of coaching: 




File Studies Uariable Analysis Charts Options Printer Info 



■: ;;'i I • i ; i ;t ;; 



ji; 

tv, ic , , -i , . :t 

lii 1 III 

iliiliilHi iiilij 



‘•'V 







EDiTVARIf^LEbEFiHltlbii 



' =•- s • J ■ 

lifillilij 



Mo: -11 
type: C 



ID: HOURS Descr : Nura of hrs of coaching 

length: -6 decimal places: -2 



B character ahreuiation of uariable 



, lilt,* 

V^Fl-Help 



- ,* 

FlO-Menu 



Fl-Help F2-UARS F10-Contiime ESC-quit 






i 1 1 j I'l iji , ' I j » i 

iiiiii :>:>!; 

i < i ■ 

m mm 

■ j 

i-i'l; i'i'i'i iVi- ii 

vo : ;;;yH ; iyr liji 

!;iH vl'-j'jy 

•i ,v, : . .ill-. ,’l 
i i 5.1 i i'i I'lj 
i ;V: i i ;Y,: Vi 





i ; i i ; ; vi : i ;'i : i : 






iiiiiiiiiiiisiaiiiiiiiiiii!! 



mmm 

ifijiiii Ililli 

. , i ’ i l i i » i i'i ij i'i 

mmsm 

i-i. . s 1 i 

iiisi 



COACH0 



23:53:18 



BEST COPV AVAILABLE 

Chapter 1 



18 



5 






Introducing Meta-Analysis and Meta-Stat 



Meta-Analysis User’s Guide 



You can also create vanables to track the characteristics of the studies in your 
meta-analysis. For example, suppose you want to analyze the relationship 
between study outcomes and study quality. You can create a variable to identify 
the quality of each study. In this example, the variable can identify up to three 
levels of quality: 
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After creating the variables you need, you use the variables to code the features 
of each study. The following example shows the entry screen you use to enter 
data about each study: 
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Transforming Study Results into Effect Sizes 

In addition to coding the features of each study, you must also code the results. 
Because results can be reported in different ways in different studies, Meta-Stat 
allows you to enter the outcome statistics as reported in the original studies. 

Meta-Stat transforms the outcome statistic into the Effect Size. The Effect Size is 
the standardized difference between the means of the treatment group and the 
control group or between the means of two treatment groups. You can read more 
about the Effect Size in Chapter 6. 
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Some study results are more properly described with correlational coefficents, 
differences between percents, or even odds ratios. Meta-Stat can also analyze 
correlations and has routines for correct for several attenuating factors. The plain 
statistics option can be used for analyzing other types of data. 

In the following example, Meta-Stat will calculate the Effect Size from the F 
Statistic reported in the original study. 



File Studies Variable Analysis Charts Options Printer Info 



TUDV ID: ALLEN TITLE: Effectiveness of study counseling 
TUDY NUK: 1 of 31 AUTHORS : Allen, G. J. 



apmerezgffect Size! , ",i V ' 1 , ! , , • i 

;, i :z i :ufiBifisED,:junbiased Effect size; : s ; : 1 1 i*:,:;;!; .v 






(1 - (3 / (M « TOTAL M) - 9))) » EFFECTSZ" v,w - .0.909 
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RR F Statistic: 

Degrees of freedom: 

Direction: positive 
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Applying Statistical Techniques to Your Data 

After you enter the coded study features, Meta-Stat can help you uncover the 
secrets in your data by applying a full set of statistical techniques. 

You can display simple descriptive statistics. This example shows Descriptive 
Statistics for the Effect Size variable from a particular meta-analysis: 



HETA-STAT 



09/10/33 



DESCRIPTIVE STATISTICS 



Criterion Variable: 
Ueighting Variable: 
Select If: 



UNBIASED Unbiased Effect Size 



Number of Valid Observations: 
Number of Hissing Observations: 

unweighted 



Mean 


0.591 


Median 


0.479 


Variance 


0.376 


Stand. Dev. 


8.613 


Nin Value 


-0.287 


Nax Value 


2.087 


Range 


2.374 


Hedges's Hi 


116. 965 



29 

1 



(p = 0.800) 
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You can obtain a host of statistics that are described in the meta-analysis 
literature as show on the following screen: 



KETA-STAT 



09/10/93 



DESCRIPT IUE STATISTICS 



Criterion Uariable: 
Weighting Uariable: 
Select If: 



UNBIASED Unbiased Effect Size 



Number of Ualid Observations: 
Number of Hissing Obseruat ions : 



29 

1 



unweighted 



Hean ES 
Fisher's Z 
Equiu ES 
Equiu R 
Equiu R SQ 
Failsafe N 
Uar Ratio 



0,591 

1.497 

0.579 

0.270 

0.677 

0 

3.750 



(p=0 .13) 



|^Ctrl-P Print F1Q Menu tl next uariable PgUp basic 
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Using Regression analysis, you can perform regression calculations using any 
predictor variables that you choose: 



f KETA-STAT 09/10/93 
DEGRESS IOM 

Criterion Uariahle : UNBIASED Unbiased Effect Size 

Predictor Uar tables: HOURS 



Weighted by: 

Llstulse deletion 
Solution took 1 iterations 


WEIGHT 


Inuerse of the 


variance 




Multiple R 
Adjusted R-Square 


0.7706 

0.5776 


R Square 
Standard 


0.5939 
Error 1.3637 






DF 


Sun of 
squares 


Mean 

square 


P 


Regression 

Residual 


1 

25 


67.9078 QCR) 
46.4942 QCE) 


67.9078 

1.0598 


0.0068 

0.0001 


Ctrl-P Print F10 Menu 




PgDn More 
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You can study the differences between groups. In the following example from 
the Group Means analysis, Meta-Stat has calculated the Effect Size mean, standard 
deviation, and 95 percent confidence intervals for a grouping variable called 
Publication Type. This variable identifies where a study was published: 



META-STAT 



Criterion Uariable: UNBIASED 



GROUP MEANS 



publication 


Mean 


St. Deu. 


N 


952 Confid 


Interval 


60-76 


1.146 


0.413 


5 


0.671 to 


1.620 


71-75 


0.532 


0.534 


5 


-0.601 to 


1.146 


76-00 


0.557 


0.673 


15 


0.107 to 


0.920 


01+ 


0.097 


0.260 


4 


-0.264 to 


0.45B 



Hedges's Homogem ty Tests 

Source Statistic df 

Total 116.965 20 Cp = 0.600) 

Within 97.591 25 (p = 0.600) 

Between 19.373 3 (p = 0.600) 
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You can also tell Meta-Stat to create dummy variables from a grouping variable. 
This allows you to explore the differences between the groups identified by that 
variable. The following example shows regression results for which dummy 
variables were created from the variable Publication Type. The results allow you 
to compare studies based on where the studies were published: 



f UTA-STAT 03/10/83 ^ 

i REGRESS ION 1 



Uariable 




Beta 


B 


St Err 


t 


P 


PUB TV GP 


journal 


0.212 


0.281 


0.344 


6.817 


0.580 


PUB TV GP 


d issert 


0.307 


0.451 


0.351 


1.285 


0.137 


FUBTV GP 


paper 


0.266 


0.325 


0.355 


0.314 


0.636 


PUBTV.GP 


other 


0.462 


1.377 


0.426 


3.230 


0.002 


Constant 






0.588 


0.414 


1.226 


0.213 



Standard errors follow Hedges and Olkin (1385. pages 167-188). 



Ctrl-P Print F10 Hem 



PgUp .PgDn More 22:21:26 









Chapter 1 



26 



13 




Introducing Meta-Analysis and Meta-Stat 



Meta-Analysis User’s Guide 



Displaying Results 

Meta-Stat can display your data on a variety of charts. For example, you can 
show the Effect Size and 95 percent confidence intervals for each study: 
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2 Installing and Setting Up 
Meta-Stat 



This chapter gets you started with Meta-Stat. It shows you how to install the program 
and fine tune it for your equipment and tastes. 



Getting Started with Meta-Stat 



Installing Meta-Stat 

Meta-Stat ' s simple installation process is completely automated. All you need to 
do is tell Meta-Stat the disk drive and directory where you want to install the 
program. Follow these steps: 

1 . Click on the Download Meta-Stat link on the Internet. 



You will the following screen 



File Download 





You are downloading the file: 
metasetup.exe from 1 28.8.1 82.4 



Would you like to open the file or save it to your computer? 



Save Cancel More Info 



wey; ask before opening thisV/ps of tils 
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2. Click on Open. The windows-based installation program will then run. 

3. Follow the instructions on your screen. We encourage you to accept the 
default value. This will create links on your start menu and on your desktop 

4. Meta-Stat can be un-installed from either the Start Menu or from the 
Control-Panel - add/remove software program. 

NOTE: Meta-Stat uses a .PLF file. They key settings are to run the program 
using full screen and to enable quick-edit. 



Starting Meta-Stat 

Follow these steps to start Meta-Stat. 

1 . Click on the Meta-Stat icon on the desktop 
OR 

2. Click on the Meta-Stat icon in your Programs directory (Start / Programs / 
MetaStat/ Meta Stat ) 

OR 

3. From the DOS prompt, change to the drive and directory where you installed 
Meta-Stat. and type META to start Meta-Stat. 
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You should see the Meta-Stat main screen: 




NOTE: Use Meta-Stat in full-screen mode rather than a smaller window. 

This will ensure proper mouse and graphic functioning. Push Alt- 
Enter from windows to change screen size. 



Start-Up Options 



You can use two options when you start Meta-Stat. 

• Use META filename to start Meta-Stat and open the meta-analysis 
stored under filename. If you do not use this option, you must use 
Hie/Open to open a meta-analysis once you start Meta-Stat. 

• Use META -m to start Meta-Stat in monochrome mode. 
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Exiting from Meta-Stat 

You can exit from Meta-Stat in two ways: 

• Open the File menu and select Exit 

To find out how to work with menus, see "Using the Meta-Stat Menus" 
later in this chapter. 

• Press and hold down Ctrl, then press X. 

If you have made changes to the current meta-analysis, Meta-Stat prompts you to 
save your changes. 



Using the Meta-Stat Menus 

Meta-Stat offers pull-down menus from which you can select options. 



General Menu Characteristics 

• After you start Meta-Stat , your first keystroke opens the File menu, 
because the program assumes you want to open or create a meta- 
analysis. 

• If a menu option is not highlighted, you cannot select it. 

For example, if you have not yet opened a meta-analysis, the Save and 
Save as options on the File menu are not highlighted because you do not 
yet have anything to save. 

• Press F10 to get back to the menus after you finish working with a menu 
option. 

• Open or close the menus with either the keyboard or the mouse. See the 
following sections for details. 
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Controlling the Menus with the Keyboard 

• You can close the menus by pressing Esc. 

• Press - and - to highlight menus to the left and right, respectively. 

• If the menus are closed, you can pull down a menu either by: 

Typing the first letter of the menu name, or 

Highlighting the menu name by pressing the - and - cursor keys 
until the name is highlighted, then pressing Enter. 

For example, suppose you have closed the menus by pressing Esc, as 
shown here: 
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You can now open the Analysis menu by typing A, or by highlighting the 
menu name and pressing Enter or left-clicking: 




• After opening a menu, you can select an available option by: 

Typing the highlighted letter of the option name, or 

Highlighting the option by pressing the cursor keys until the 
option is highlighted, then pressing Enter. 
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Controlling the Menus with the Mouse 

• The mouse works in a way similar to the keyboard, except that you use 
the left button for Enter and the right button for Esc. 

• You can close the menus by right-clicking. 

• Move the mouse to highlight menus to the left and right, respectively. 

• If the menus are closed, you can pull down a menu by moving the mouse 
until the menu name is highlighted, then left-clicking. 



Using Quick Keys for Menu Options 

Meta-Stat uses Ctrl to provide quick-keys for some menu options. You can use 
the quick-keys to select these options from anyplace within Meta-Stat. You do 
not have to pull down the appropriate menu first. 

For example, you can press Ctrl-Q to quit from anywhere within Meta-Stat. You 
do not first have to pull down the File menu, where the Quit option is listed. 
When you use a quick-key, you must press and hold down Ctrl, then press the 
letter key. 

The quick-keys are listed on the menus and in the chart below. The A symbol is 
used to represent the Ctrl key. 



Quick-Key 


Description 


A S 


Save your data 


A X 


Quit Meta-Stat 


A R 


Recompute the values of all variables in the meta- 
analysis that are based on equations 


A N 


Create a new variable 


A E 


Edit a variable 


A K 


Kill any print jobs that are currently printing 
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Configuring Your Display 

Meta-Stat offers a variety of display options. You can display Meta-Stat screens 
in color or monochrome, and you can select from three screen sizes. For 
example, you can choose to display 50 lines on your screen, making a large 
amount of data visible at once. 

You can also turn sound effects on and off, and you can refresh the screen to 
make it more readable. 

The following sections display your options. 

Setting Display Options 

1 . Select Options/Colors. 

2. Highlight either Color or Monochrome and press Enter. 
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3 . Select Options/scrEen Size 

4. Select the number of lines you want to display — 25, 43, or 50. Make sure 
that your display equipment is capable of the resolution you select. 

5. Select Options/Save Config to save your display settings. 



Turning Sound Effects On and Off 

When you make an error, Meta-Stat beeps. You can turn off this sound if it 
annoys you. 

Select Options/sound eFfects. Meta-Stat removes the check-mark next to this 
menu option, to indicate that it is turned off. 

Select the option again to turn the sound back on. 



Refreshing the Screen 

As you use Meta-Stat and create studies, modify variables, and enter data, the 
screen might become garbled. Nothing is wrong with your data; only the screen 
display is incorrect. 

To refresh the screen, select Options/Refresh Screen. 



Configuring Your Mouse 

When you create graphs with Meta-Stat , you can use your mouse to point and 
click on a data point to display more information about the data. If your graphs 
contain a large number of data points, you can make your mouse more sensitive, 
allowing you to fine tune the mouse's location on a data point more precisely. 
See Chapter 8 for more information about graphing. 

NOTE: Meta-Stat does not include basic mouse-driver software. This 

software, which came with your mouse, must be loaded before you 
start Meta-Stat. Consult your mouse documentation for more 
information. 
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To control the sensitivity of your mouse, follow these steps : 

1 . Select Options/ Mouse Speed 

Meta-Stat prompts you to identify the mouse sensitivity in terms of 
horizontal and vertical units. These units are the number of screen pixels the 
mouse pointer moves at one time. 




2. Type the number of pixels you want the pointer to move. Larger numbers 
cause faster but less precise movement, because the mouse pointer jumps 
more pixels with each movement. 

3. Press F10 when you have finished. 

4. Select Options/Save Conflg to save your mouse settings. 
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Setting the Default Directory 

When you work with meta-files, Meta-Stat assumes you want to get files from 
and save them to the directory in which Meta-Stat was installed. You can change 
the directory that Meta-Stat uses as a default. 

Follow these steps: 

1 . Select File/cH Directory 

2. Meta-Stat displays a list of directories in the Meta-Stat directory. In the 
following example, there are two directories, COACHING and DRILLS: 



File Studies Variable Analysis Charts Options Printer X Info 
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3. Use the cursor keys or your mouse to move the highlight to the directory 
you want to use, then press Enter. 

To select a directory that is at a higher level in the directory structure, 
highlight the .. and press Enter. 

4. To tell Meta-Stat you have finished, highlight the . and press Enter. 
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Printing with Meta-Stat 

Meta-Stat provides a variety of printing options. You can print variable 
definitions, studies, statistical analyses, and even graphs. The following sections 
describe how to configure your printer and print with Meta-Stat . 



Setting Up Your Printer 

You must identify your printer to Meta-Stat. You can do this at any time. Follow 
these steps: 

1 . Select Printer/Select Printer 

Meta-Stat displays a list of printers that it supports: 
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2. Scroll through the list, by pressing 1 , until you have highlighted your printer. 

NOTE: If you do not see your printer in the list, select a printer that is 

compatible with yours. 

3. Press Enter when you have finished. 

4. If your printer can print output in both portrait and landscape orientations, 
use Printer/Orientation to select the orientation you want to use. 

5. If your printer can print in color, use Printer/ Colors to control color printing. 

6. Use Printer/ Printer Port to identify the port to which your printer is attached. 

NOTE: If your printer is attached to a COM port, Meta-Stat asks a series 

of questions about the port's communication settings, including the 
flow rate, stop and start bits, and parity. For most situations, the 
default settings should work properly. 

7. Select Options/Save Config to save your printer settings. 



Printing to Paper or Disk 

You can print output on your printer, or you can route the output to a disk file. 
You might route output to a disk file if you need to share it with other 
applications, or if your printer is not currently available. 
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1 . Select Printer/Destination. 

Meta-Stat asks you where to send printed output: 




2. Select the destination you want to use. 

File Names for Disk Files 

If you select Disk as the destination for printed output, Meta-Stat saves the 
output in a disk file: 

• In the Meta-Stat directory 

• Under a file name of PRINTER 

• With an extension of 001 for the first printer-output file, 002 for the 
second, and so on. 
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For example, PRINTER.001 is the name of the first printer-output file that you 
create. PRINTER.012 is the name of the twelfth. 

You can delete these files if you no longer need them. 



Spooling Print Jobs 

Meta-Stat has an exceptional built-in print spooler which will print your work as 
a background task. In most cases you will want to turn this option on. If you are 
on a network or are having printing problems, try turning this option off. 



Killing Print Jobs 

Use Printer/Kill print jobs to stop spooling output to the printer. Once a file or 
graph has been spooled, this option is no longer effective. 
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Traditional, narrative reviews of research literature are selective. Critics of such studies 
rightly ask, "Why were some studies selected but not others? What criteria were used? 
What kind of bias did the selection process introduce into the review?" 

A meta-analysis should be comprehensive and replicable. It should not only examine as 
much of the research as possible, it should also describe how you found the research so 
that other researchers can evaluate your work. 

This chapter describes the tasks required to organize and conduct a meta-analysis. 



Focusing Your Aim 

Before setting out to collect studies, you must first decide the range of topics 
your review will encompass. Meta-analyses typically cover broad topics, 
sometimes loosely defined topics. They examine questions such as, Does 
psychotherapy work? Does computer-assisted instruction lead to more learning 
than traditional instruction? Is mastery learning better than traditional learning? 

These questions, by themselves, are too broad to be meaningful. Does 
"psychotherapy," for example, mean reading a self-help book, or spending a few 
sessions with a college counselor, or completing a multi-year psychoanalysis? 
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To avoid vague generalities, you must make the focus of your meta-analysis 
much more explicit by establishing criteria for including or excluding studies. 
However, the criteria cannot be too restricting, or you might not find a sufficient 
number of studies; and even if you do find enough studies, you might not find 
anything interesting or illuminating. 

To focus your aim, read a good sample of the literature and develop a thorough 
understanding of the concepts and methods that you want to analyze. Determine 
the effect, or outcome, you want to study, as well as the predictors, or 
independent variables, that you want to measure. 



Conducting the Literature Search 

A thorough literature search is critical to the validity of your meta-analysis: 

How one searches determines what one finds; and what one finds is the 
basis of the conclusions of one's integration of studies. (Glass, 1976) 

Finding research studies is difficult and time-consuming. The studies can be 
located in a variety of places, and often you must look beyond the titles. For 
example, if you are conducting a meta-analysis of sex differences, you will find 
that some studies show such differences even when that is not the principal 
variable of interest in the study and even when the study title makes no mention 
of sex differences. 

To find studies, check the following general categories described by Rosenthal 
(1984): 



• Books 



- Authored books 

- Edited books 

- Chapters in edited books 

• Journals 

- Professional journals 

- Published newsletters 

- Magazines 

- Newspapers 



32 



44 



Chapter 3 




Meta-Analysis User’s Guide 



Setting Up a Meta-Analysis 



• Theses 

- Doctoral theses 

- Master’s theses 

- Bachelor's theses 

• Unpublished work 

- Technical reports 

- Grant reports 

- Convention papers not published in proceedings 

- ERIC reports 

- Films and cassette recordings 

- Other unpublished work 

Also consider soon-to-be-published works. Given the time lags between data 
collection, wnte-up, journal acceptance, and publication, the latest word on a 
topic may well be in the pipeline to publication. Your literature search should 
produce a list of names you can contact with inquiries about recent research. 
These researchers will also know about others who are doing research on similar 
topics. 

Last, and perhaps most important, consult secondary sources such as review 
periodicals and the myriad abstract archives that are available in most fields of 
study. These sources will help you identify the primary research that will be the 
subject of your meta-analysis. 



Handling Questions of Study Quality 

One of the most controversial questions related to meta-analysis is the question 
of whether to include studies that are of doubtful or poor quality. Some critics 
invoke the Garbage-In, Garbage-Out principle by arguing that any meta-analysis 
that summarizes studies of widely differing quality is likely to be uninformative 
or flawed. These critics argue that studies with methodological flaws should be 
eliminated from consideration in the meta-analysis. 

Others counter by noting that it is often difficult to assess methodological quality 
and researchers often disagree on quality. Despite a researcher's best attempts to 
provide an objective measure of quality, decisions to include or exclude studies 
introduce bias into the meta-analysis. 

Others note that the quality of a study may not have an effect on the study's 
outcome. When in doubt, include the study in the meta-analysis and use an 
independent variable to code the quality of a study. Then examine empirically 
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whether the outcome does in fact vary with study quality. You can do this with 
Meta-Stat by examining the relationship between study quality and Effect Size. 



Dealing with Publication Bias 

Well-recognized publication biases can produce bias in your meta-analysis. 

First, publication policy is biased toward statistically significant findings. In 
other words, it is easier to get published if you have something statistically 
significant to report. A number of meta-analyses have indeed found that effect 
sizes reported in journals differ widely from unpublished work. The Effect Size is 
about 33 percent larger in published research. 

Second, publication biases can skew the direction of effect. For example, Smith 
(1980) reported that, in the published literature, counselors and therapists tend to 
view women more negatively than men. In the unpublished literature, the 
opposite is true. As Strube (1985) theorized: 

Such biases probably reflect the fact that some findings better "fit" the 
prevailing scientific atmosphere (Zeitgeist) and are scrutinized less closely 
than are novel or counterintuitive results. 

To overcome publication bias, gather everything you can find, both published 
and unpublished. You can then use empirical methods to examine the question of 
publication bias. For example, with Meta-Stat you can compare the effect sizes 
for published research and unpublished work. 



Documenting Your Search 

A traditional, narrative review of literature often devotes only one or two 
paragraphs to a discussion of the methods that were used to conduct the 
literature search. A meta-analysis requires a much more rigorous approach. Only 
through a comprehensive description can other researchers evaluate the validity 
of your work and, perhaps, gather clues about overlooked sources. 



34 



46 



Chapter 3 




Meta-Analysis User’s Guide 



Setting Up a Meta-Analysis 



Creating the Meta-Analysis in Meta-Stat 

Before you can define your variables in Meta-Stat , as described in the following 
sections, you must create the meta-analysis in Meta-Stat. Chapter 4 of this 
manual describes how to create and modify the set of data files that, together, 
make up a meta-analysis. See Chapter 4 for complete information . 



Defining Variables 

After you collect the literature for your meta-analysis, you must determine which 
study features your meta-analysis will examine. These features become the 
variables in the meta-analysis. 



Classes of Variables 

There are three classes of variable: 

• Variables that identify characteristics of the study 

You will need some variables to identify when and where the research 
was published, whether a control group was used, the type of effect that 
was measured, and so forth. 

These variables will help you show the relationships, if any, between 
study methodology and results. 

• Variables that identify characteristics of the sample 

Use these variables to identify the subject population — age, educational 
level, socioeconomic status, and so on. 

• Variables that identify characteristics of the Intervention 

These variables could include the type of treatment or intervention, its 
duration, the type of effect that was measured, and so on. 
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Variables for a Meta-Analysis of 
the Effects of Psychotherapy on Asthma 

Schlesinger et al. (1978) conducted a meta-analysis that evaluated the effects of 
psychotherapy on asthma. This meta-analysis found that psychotherapy does 
indeed significantly lessen both the medical and nonmedical effects of asthma. 

In the meta-analysis, the following variables were used: 



This variable ... 


Was used to identify ... 


Therapy Type 


Type of psychotherapy received by treatment 
subjects 




For example, some subjects received 
hypnotherapy while others received group 
therapy. 


Age 


Average age of subjects 


Hours of Therapy 


Number of hours that subjects received therapy 


Control Group 


Type of therapy received by the control group 




Some control subjects received no treatment, 
others received relaxation therapy, and others 
received medical treatment. 


Follow-Up Time 


Amount of time between the end of treatment 
and the measurement of the outcome variable 


Dependent Variable 


The type of outcome or effect that was 
measured 




For example, some studies measured the effect 
of psychotherapy on the use of drugs to treat 
asthma; others measured the effect on asthma 
attacks and hospitalization. 


Effect Size 


The size of the effect of treatment 



Variables for a Meta-Analysis of 
the Effects of Coaching on SAT Scores 

Becker (1990) conducted a meta-analysis that tested the effects of coaching on 
Scholastic Aptitude Test scores. Among other results, this meta-analysis found 
that the effectiveness of coaching varied widely across the studies, with much of 
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the variation resulting from studies without comparison groups. The magnitude 
of the coaching effect is related to study design and to the duration of coaching 
intervention. 

The following table describes some of the variables that were used in the meta- 
analysis: 



This variable ... 


Was used to Identify ... 


Year of publication 


Year the study was published 


Type of publication 


Whether the article was published in an 
academic journal 


ETS Authorship 


Whether any author was affiliated with 
Educational Testing Service, which develops 
and administers the SAT 


Control Group 


Type of control group 


Use of 

randomization 


Whether subjects were randomly assigned to 
control and coaching groups 


Selectivity 


The type of students in the study (e.g. low 
achievers, public-school students, college-prep 
students) 


Voluntariness 


Whether participation in coaching was voluntary 
or compulsory 


Duration 


Length of coaching program 


Presence of Test 
Practice 


Whether students practiced taking complete 
sample tests 


Homework 


Whether students were given coaching-related 
exercises for completion at home 


Use of Computerized 
Instruction 


Whether students were given computerized 
practice 



Deciding Which Variables 
to Use for Your Meta-Analysis 

In some cases, simple analysis will help you decide which variables are required 
for a meta-analysis. But it's not always so simple. Definitions of study features 
may vary from study to study, and different researchers often use different 
terminology. 
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The safest procedure is to have two researchers independently code all study 
features, then negotiate an agreement in places where they disagree. 



Setting Up Variables in Meta-Stat 

Chapter 5 describes how to create and modify variables in Meta-Stat . The 
program comes with many variables already predefined for you. There are 
variables to measure: 

• Study characteristics, including the year of publication and the type of 
publication in which the research appeared 

• Sample characteristics, including the sizes of the treatment and control 
groups 

• Intervention characteristics, including the Effect Size and the Unbiased 
Effect Size 

You can also create your own variables to handle other study features. You can 
create different types of variables to handle different types of data. See Chapter 
4 for complete information about Meta-Stat variables. 



Coding the Results of Each Study 

After defining the variables you will use to measure studies, you must obtain the 
studies and code data for each variable. 

Unfortunately, this task is not as straightforward as it might seem. Meta-analysts 
sometimes find that the required data are missing and must be estimated. 
Sometimes, researchers find errors in the original research. Errors in the 
publication are relatively easy to correct. But that is not the case for errors that 
were made when the data were originally recorded. These errors will 
contaminate your meta-analysis, but it is no easy matter to determine whether 
that contamination is severe. 
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Dealing with Data Errors 

In social science research, errors commonly occur in favor of the anticipated 
outcome. In one study, researchers gave genetically identical white rats to 
students. The researchers told the students that some of the rats were fast runners 
and some were slow. When the students recorded running times in a maze, the 
"fast" rats had faster times than the "slow" ones. 

Another study collected the original data from 27 published studies. The median 
error rate was about 1 percent, but rates of 3 percent and 4 percent were not 
uncommon. Two-thirds of these errors supported the researcher’s hypothesis. Of 
course, if the researchers were unbiased, half of the errors should have supported 
the hypothesis and half should have countered it. 



Dealing with Data Reporting Deficiencies 

The scientific method requires researchers to report their studies in enough detail 
that anyone who wants to replicate a study can do so. Unfortunately, published 
research seldom exhibits the required clarity and detail. Further, there is no 
standardization among different journals — or even within a journal, as editors 
change — about what all studies must contain. About 40 percent of all studies 
even lack the means and standard deviations that are important to most meta- 
analyses. Fortunately, this deficiency does not make a meta-analysis impossible. 

Deficient reporting can affect your meta-analysis by forcing you to exclude 
variables that, although potentially important, are not described in the study 
results. Deficient reporting can also force you to abandon precise variables in 
favor of rougher groupings of data. This problem causes less variance in your 
variables and attenuates important relations in your data. 

In your meta-analysis, it may be useful to add a variable that you can use to rate 
data quality. If you rate the quality of each study's data on a three-point scale, 
you can then examine whether the quality of reporting systematically affects the 
outcomes of studies. 



Entering Study Data into Meta-Stat 

Chapter 6 describes how to enter each study into your meta-analysis. For each 
study, you must identify the author and title. You must then enter data from the 
study for each variable that you have set up. See Chapter 6 for full information . 
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Analyzing and Displaying Results 

Chapters 7 and 8 describe how to analyze and display the results of your meta- 
analysis. 

Chapter 7 describes the statistical techniques you can use to explore your data. 
Chapter 8 describes the graphs you can use to analyze and display results. 



First Steps 

The meta-analyst is well advised to perform a number of very basic analyses 
before launching into more complex analyses in which some of the more basic 
features of the data base are obscured. Specifically, after entering the data, it is 
wise to perform simple frequency distributions and scatter plots to see if data 
have been entered correctly or whether some studies contain data that are clearly 
aberrant in the context of all studies in the analysis. These aberrations can occur 
in a variety of ways. Most simply, perhaps, are clerical errors committed in the 
entry of the data into Meta-Stat. Bad data points can enter the process earlier: 
typos in printed reports of the studies; data analysis errors in the primary data 
analysis that yield an F-ratio in error by a factor of two or three, for example; or 
conditions invalidating the transformation of a reported statistic into an Effect 
Size or correlation coefficient. However they arise, these errors produce odd 
values of variables that can distort the analysis of the study outcomes and their 
relationships with study characteristics. The errors need to be detected and 
corrected or removed. 

Two analyses are particularly useful to this end. Under CHARTS can be found the 
options of BAR GRAPH and EFFECT SIZE BY STUDY. An early analysis should be the 
graphing of EFFECT SIZE (or CORRELATION in the case of correlations as the study 
outcome) as a BAR CHART. The results are then inspected for a small number of 
entries that are clearly separated from the bulk of the distribution of effects. How 
far removed from the mean or the next closest entries is "clearly separated"? No 
definitive answer can be given. Values more than five standard deviations from 
the mean in samples of 100 or fewer cases would certainly raise suspicions about 
errors in reporting or calculation. But it is difficult to be more precise than this 
here. (The reader interested in pursuing this point to a more precise answer 
should consult Dixon & Massey, 1969, or Tukey, 1977.) 

The graph of EFFECT SIZE BY STUDY can add information to the BAR GRAPH. All 
effect sizes are arrayed in either ascending or descending order and each is 
bracketed by a confidence interval that reflects on the size of the samples in the 
study yielding the Effect Size. If the largest or smallest of these effects are very 
distant from their neighbors and the confidence intervals are likewise quite 
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distinct and not overlapping, then it might be well to return to the studies that 
produced these effects and see if some error is apparent. Not every aberration 
will have a simple explanation; when they lack reasons, the analyst is faced with 
the difficult question of eliminating the odd data point without good reasons or 
leaving it in and having it exercise undue influence on the later statistical 
analyses. 

The problem of detecting outliers can be more complex than indicated above. 
Consider the follow data, which for the sake of example can be thought of as 
Effect Size from nine experiments in which a drug is tested against a placebo: .40, 
.45, .50. .55, .60, .75, .80, .85, .90. There appear to be no outliers or aberrant data 
points here. But suppose that the first four and the ninth effect sizes are from 
studies that are double blind and the fifth through the eighth effects are from 
single blind experiments: when the effects are grouped and inspected, the value 
of .90 appears aberrant and sends the analyst looking for errors or explanations: 

Single Blind Effects: .60 .75 .80 .85 

Double Blind Effects: .40 .45 .50 .55 .90 

Consequently, it is advisable not only to inspect the entire data set for outliers 
but to break down the whole data set into various groups to see if aberrations 
appear when effects are compared with effects observed under similar 
conditions. Perhaps the best way to accomplish this analysis is to select EFFECT 
SIZE BY STUDY under the CHARTS menu and then use SELECT IF to construct 
selection criteria to view the effects for various subgroups of discrete study 
characteristics. 

For continuous study characteristics, the counterpart to the above analysis is a 
scatter plot relating the study characteristic to the study outcome (e.g., EFFECT 
SIZE). When SCATTER PLOT is selected from the CHARTS menu and EFFECT SIZE 
plotted against a study characteristic, a quick visual fitting of the relationship 
will reveal points that fall far away from the general relationship between the 
two variables. The studies contributing these outliers can be identified with the 
POINT SHOOT feature, and reread or checked for obvious errors. 

An alternative means of detecting bad data points is to perform REGRESSION 
analyses under the ANALYSIS menu and inspect the residuals graph that the 
analysis produces. Very large and aberrant residuals identify data points that 
have peculiarities unrelated to the study characteristics used in the regression. 

Although outliers can be critical when they occur in the study outcome variable, 
say EFFECT SIZE, they can occur in other places in a data set. Occasionally they 
cause problems in subsequent analyses. The methods suggested above for 
searching for outliers in study outcomes can also be applied to any other variable 
(with the exception of the EFFECT SIZE BY STUDY graph that is only available for 
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the Effect Size variable). In addition, the BREAKDOWN tables under ANALYSIS are 
useful in detecting bad data points in Grouping Variables that describe study 
characteristics. 



A Comment on Schmidt and Hunter Meta-Analysis 

The effect-size meta-analysis approach in Meta-Stat tends to follow the concepts 
outlined by Glass and by Hedges. These techniques are very similar to those of 
Schmidt-Hunter, but they do differ in several critical ways. Schmidt and Hunter 
argue that: 

1) the Effect Size should be based on the pooled variance estimate rather 
than that of the control group, 

2) the meta-analyst should correct for sampling error, and 

3) the meta-analyst should correct for measurement error. 

The argument for using the pooled variance estimate is that it is based on more 
observations and subject to less error than the control group variance estimate. 
The counter argument is that the control group variance is unaffected by the 
treatment. The effect-size calculator in Meta-Stat lets you choose your 
technique. In practice there will be very little difference. 

Schmidt and Hunter recommend grouping, trimming, or selecting your data until 
the ratio of the error variance to the variance of the effect sizes is .75 or greater. 
They argue that if 75% of the variance is due to error, then the rest of the 
variance is probably also due to error. Therefore the population variance is zero 
and the model of a single Effect Size is consistent with the data. Meta-Stat 
provides you with their variance ratio under Analysis/Descriptive when the 
criterion variable is UNBIASED. 

Their last key recommendation is to correct for study artifacts such as the lack of 
perfect reliability of the criterion measure and data dichotomization. For 
example, they recommend correcting for measurement error by dividing the 
effect-size estimate by the square root of the reliability of the criterion variable. 
They note that measurement error inflates the standard deviation and thus lowers 
the Effect Size. Meta-Stat can accommodate this correction. The program allows 
you to override values entered into an equation when one or more of the 
variables are missing. Thus, you can correct for measurement error and other 
artifacts by following these steps. 

1 . Use the Effect Size calculator to compute EFFECTSZ. 
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2. Manually divide the EFFECTSZ (or UNBIASED) value by the square root of 
the dependent measure reliability (reliability for commercial tests can be 
found in the Buros Mental Measurements Yearbook). 

3. Divide the above value by other correction formulas if desired and 
available. 

4. Delete the value for EFFECTSZ. 

5. Insert the corrected effect-size in UNBIASED. 

Not all meta-analysis methodologists will agree with each of Hunter & Schmidt's 
recommendations. We diverge from their plan in at least three respects. It seems 
unwise to contaminate the variance estimate with the treatment effect. 
Furthermore the 75% rule-of-thumb test is quite arbitrary and not in the spirit of 
intelligent, contextualized data analysis. Third, corrections for measurement 
unreliability often make conceptual sense but are impossible to implement. 
Finding a test reliability (in a book or manual or from a computer printout) is one 
thing; but assessing a sensible and appropriate error variance is quite another. 
The error that enters the control group variance estimate and contributes to 
observed differences among persons is generally not the error assessed by 
ordinary convenient measures of test reliability like Cronbach’s Alpha, 
Kuder-Richardson or various split-half coefficients. In addition, after reanalyzing 
several meta-analysis studies using the Glass and the Schmidt-Hunter 
techniques, Hough and Hall (1991) demonstrated that the corrections are trivial 
and that both approaches lead to the same conclusions. 
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Summary of Steps for a Meta-Analysis 

To conduct a meta-analysis: 

1 . Focus your aim by deciding on an area of the literature that you want to 
explore and a specific topic that you want to analyze. 

2. Conduct a thorough literature search. 

3. Create the meta-analysis in Meta-Stat. See Chapter 4. 

4. Add variables that you will use to code study features. See Chapter 5. 

5. Read the studies and code the variables for each study. 

6. Enter the data for each study into Meta-Stat. See Chapter 6. 

7. Explore and display the data with various statistical techniques. See Chapter 
7 and 8. 
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This chapter describes how to open and save your meta-files, and how to share the data 
in a meta-file with other applications, such as SPSS. 



Creating a Meta-file 

Your first task in performing a meta-analysis is to create a meta-file. A meta-file 
holds the following information: 

• Variables that you will use to conduct your meta-analysis 

• Studies that you will analyze 

A meta-file is actually two DOS files . For more information, see the section 
" Keeping Backups of Your Data " later in this chapter. 

Follow these steps to create a meta-file: 

1. Select File/New 

2. If necessary, close any open meta-file by following these steps: 

• If you have already opened a meta-file, Meta-Stat asks if you want to 
close it. Press Y or N. 
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• If you tell Meta-Stat to close the open file, and you have made changes 
to it, Meta-Stat asks if you want to save your changes. Press Y or N. 

3. Type a name, up to eight characters long, for the new meta-file. The name 
should provide a good description of the meta-file, so that you can 
distinguish one meta-file from another on Meta-Stat screens. 

The name should also follow normal DOS file-naming rules: 

• Use numbers, letters, and the underline character. 

• Do not use punctuation such as the period or comma. 

In the following example, the meta-file is called SAT_TEST: 




BEST COPY AVAILABLE 
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4. Select the type of meta-analysis you are performing, Effect Size or 
Correlation. 




i 



best copy available 
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5. Meta-Stat displays the name of the meta-file, along with the number of 
variables and studies in it. 

Meta-Stat automatically creates some variables for you. See Chapter 4 for 
more information. 
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Opening an Existing Meta-File 

When you want to work with an old meta-file, you must open it. Follow these 
steps: 

1. Select File/Open 

2. If necessary, close any open meta-file by following these steps: 

3. Select the meta-file you want to open. If necessary, change the directory 
where the meta-file is stored. See the following sections for more 
information. 
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2. If necessary, press Space to display the sizes of the meta-files in the current 
directory, along with the date and time when they were last changed: 
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3. Press FI to display a list of commands available for sorting the list of files: 
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4. Press any key to return to the list of files. 

5. To select a meta-file, use the cursor keys to highlight the file, then press 

Enter. 
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Changing the Directory 

If necessary, you can change the directory from which Meta-Stat displays the list 
of meta-files. 

1 . Press Alt-G to display the parent directory: 
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NOTE: Instead of pressing Alt-G, you can display the parent directory by 
moving the cursor to the .. in the directory list and pressing Enter. 

2. In the list of directories, use the cursor keys to highlight the directory, then 
press Enter. 

3. In the directory you selected, pick the meta-file you want to open. See the 
previous section for more information. 
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Saving a Meta-File 

When you change a meta-file by adding variables and studies, Meta-Stat keeps 
track of the changes in your computer's memoiy. However, Meta-Stat does not 
automatically save your changes on disk. Therefore, you should make sure that 
you save your changes on disk — either periodically or when you exit. 

You can save the open meta-file using its existing name, or you can save it under 
a new name. The following sections describe your options. 



Saving Periodically 

To protect yourself from losing data during a power failure or other problem, 
you should save the open meta-file periodically. 

Select Flle/Save ( A S) to save the open meta-file. 

When you use Flle/Save, Meta-Stat uses your backup files to save any changes. 
When you exit Meta-Stat, you must again tell Meta-Stat to save changes. This 
makes your changes permanent, because Meta-Stat saves them in your original 
meta-file. If you do not save when you exit, your original meta-file is not 
updated with the additions and changes made during the session. 

NOTE: You can also use the Meta-Stat timed backup feature to protect 

yourself from data losses. For more information, see the section 
"Keeping Backups of Your Data" later in this chapter. 
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Saving Under a New Name 

You can save a meta-file under a new name. Follow these steps: 
1. Select File/Save As 

Meta-Stat prompts you to name the new file: 
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Saving When You Exit 

When you exit, Meta-Stat prompts you to save the changes to your data: 



File Studies Uariable Analysis Charts Options k^Prirrter Info 

! : 7 4 Y i [ M ' V V*' ‘ 1 | ' Vj' ! 1 1 ' . ! j ' Y.Y 7 ' 1 '.Y V ! 7 \ 7.7 7 j* V ' ‘ V 1 ' ' 'ji/i' '* ; j .! Y ‘ } . ! .!» : V 7 : .* 7 :‘ i . :i • 

:• :• v * • \j • i • • * y ■ • ■ • : * • • * : * • • • • • % *■ y • y • • • • «» ••••:»• • t j • ♦ • • • • ’ ■ • • • • e :• y 

| Ijijl ijijj! i ; » j "!i ' | > j‘> 'I’.i'! ,! |* i m 1 1 1 s ij!|! iji'iSj'l }’> }’■ ijj| i" I'li* ■!< *’j >V‘ ''‘'i i * i,i,' i|! ' « ! ! lijji ;,| {i; i * • ||! j l' 1 " j![!j!jijj|lj!$j!ji| i iSjif jjj:|!ilj})!ji{ t 1 * 1 ; jjj j!j| j jljji }!j!;!| ■!; ijjjjjjl 

1 i!!i!) , — ’ : * ' !;! ' ■ 

’ C t I J I s.y - .. ; ■ I I l % i I | 1 ( |ty I j | ) • I y t l l II It l Ij 1 y J I 1 1 1 I i I 1 I I » I ill l • III ] I I I £i I I i y I| I I ! I I l I t I I ! I C y I I 4 ijl y I* 

I i! ivjjiy. ivj.jv;-. jvi ._i|. . . i;j , i ,j; : . j.jf, . , I;, i;i; j-'I'i;. ylj i 1 . i;, , 1 . i . i l;.:! , : ,;i ! i , j y.-,v, ■ i;.v. ViV.viv.j. : ! I j ■!; ; |:: j i |:iv. 

1 ... i . . . ... . 1 . ... i . . . . 1 ! . 1 

• . 1 , • • • • '< f* . 1 - ‘ ■ • ■ ■ H ‘ 

* : - 

— -—lililil: 



| •' ■ : 

• 

iv::v:.v.v;v.7.vv,v 

im 

.t! ■ 



Metafile: COACII0 



This MetaFile consists of 



- Ex it i ng Meta-Stat 



Changes ha us ; been made to the database. 
Save changes to Metafile 1 COACH0' ?; | 




1 

. . j iflflft 

£< 5*3 

: ! !i; !!!i!iiS!!|i!iP!l! M iAiiiilil'ilivDI-liiiiAiij XA !v!v! w.M! :!uij ! m y . . . liWikitll?. .! 

I'WI&MI'Im 1 !? v-‘"’ r- 11 # 

V^Fl-Help riO-Henu COflCHB lS:B2:97y/ 



If you do not save changes, Meta-Stat does not update your original files with 
any of the changes you made during the session. 

Press Y to save your changes. 
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Copying, Deleting, 

and Renaming a Meta-File 

You can do many file-management chores directly within Meta-Stat. You can: 

• Copy a meta-file 

• Delete a meta-file you no longer need 

• Rename a meta-file 



Copying a Meta-File 

1 . Select File/ Copy 

Meta-Stat prompts you to select the file you want to copy: 
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2. If necessary, first select the directory in which the file is located. For more 
information, see the section "Changing the Directory" earlier in this chapter. 

3. Use the cursor (arrow) keys to highlight the file, then press Enter. 

Meta-Stat prompts you to identify the name of the new file: 
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4. To change the directory where the copy will be located: 

a. Press F3 to display this screen: 




b. Move the cursor to the .. and press Enter. 

Meta-Stat displays a list of directories. 

c. In the list of directories, use the cursor keys to highlight the directory, 
then press Enter. 

d. Move the cursor to the . and press Enter. 

Meta-Stat again prompts you to identify the name of the new file. 
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5. Type the filename. In the following example, a meta-file called COACHO is 
being copied to a directory called COPIES. The name of the copied filed is 

C0ACH2: 
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Deleting a Meta-File 

You can delete a meta-file that you no longer need. 

Warning: Meta-Stat deletes all files associated with the meta-analysis, 
including backup files. 

Follow these steps: 

1. Select File/Delete 

Meta-Stat prompts you to select the file you want to delete: 

2. If necessary, first select the directory in which the file is located. For more 
information, see the section "Changing the Directory" earlier in this chapter. 

3. Use the cursor keys to highlight the file, then press Enter. 

Meta-Stat asks you to confirm the deletion. Press Y or N. 
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Renaming a Meta-File 

1. Select File/ Rename 

Meta-Stat prompts you to select the file you want to rename: 




2. If necessary, first select the directory in which the file is located. For more 
information, see the section "Changing the Directory" earlier in this chapter. 
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3. Use the cursor keys to highlight the file, then press Enter. 
Meta-Stat prompts you to identify the new name for the file: 




4. Use the editing keys Del, Ins, and Backspace to remove and insert characters 
in the file name. 

5. Press Enter when you have finished. 

Keeping Backups of Your Data 

When you start Meta-Stat and open a meta-file, the program makes a backup 
copy of your data. 

The following sections show you how to make more frequent backups, and how 
to recover data from backup copies. 
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Description of Backup Files 

A meta-file actually consists of two files: 

• A definitions file contains the variables you have defined. This file is 
named: 

<meta-name> .DEF 

• A second file contains study data. This file is named: 

<meta-name> .MTA 

The <meta-name> is the name you gave to the meta-analysis. The filename 
extension, DEF or MTA, is assigned by Meta-Stat. 

When Meta-Stat creates a backup copy of your data, it copies the two files 
described above and changes the filename extensions. The following table 
describes the backup files: 



Name of Backup File 


Contents 


<meta-name>.D E! 


Copy of <meta-name> .DEF 


<meta-name>. MT! 


Copy of <meta-name>MYA. 



Creating Backup Files Frequently 

Meta-Stat creates backup files whenever you start the program and open a meta- 
file. You can also create backup files more frequently, by using a timed backup 
feature. You might want to do this when you need to make numerous changes 
during a Meta-Stat session. 
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Follow these steps to use the timed backup feature: 

1. Select Ffle/Timed Backup. 

Meta-Stat prompts you to identify how often you want to make backups: 




2. Type a number from 0 to 999. This is the number of minutes after which 
Meta-Stat will create a backup of your meta-file. Select 0 minutes to turn off 
the timed back-up feature. 

3 . Press F10 when you have finished. 
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Recovering Data from Backup Copies 

Sometimes, you might want to discard the data in your current Meta-Stat data 
files and instead use the data in your backup files. For example: 

• Your current files might be damaged. 

• You might want to discard recent changes to your data by returning to 
the backup copy that was created before you made the changes. 

Follow the steps below to recover data from backup copies. The commands you 
use in these steps rename your backup files to be the current data files for your 
meta-analysis. 

1 . Quit Meta-Stat and return to the DOS prompt. 

2. Type the following command: 

rename <meta-name> .DE! <meta-name>. DEF 
Replace <meta-name> with the name of your meta-file. 

3. Press Enter. 

4. Type the following command: 

rename < meta-name> .MT! <meta-name> .MTA 
Replace <meta-name> with the name of your meta-file. 

5. Press Enter. 

6. Start Meta-Stat again and open the meta-file. 

Keeping Backups on Diskette 

To protect important data, you should keep backup copies of the data on diskette. 
If your hard disk is ever damaged, you can recover your data from the diskette. 

To keep backups on diskette, use one of two methods: 

• Use the File/Save As option to save the meta-analysis on a diskette. See 
the section "Saving Under a New Name" earlier in this chapter. 

• Copy the meta-analysis files,<meta-/iame>.DEF and <meta - 
name>MTA , to a diskette. Use the DOS COPY command to copy the 
files. 
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Sharing Data with Other Applications 

Meta-Stat includes two features that allow you to share data with other 
applications: 

• The Export feature exports your meta-analysis data to ASCII files that 
can be read by other programs.This is coupled with an Import feature 
which reads meta-analysis data from ASCII files into Meta-Stat . 

• A stand-alone utility M2SPSS.EXE which can be used to create ASCII 
data files and fully labeled include files for SPSS. The SPSS utility is 
discussed in the next section of this manual. 

An ASCH file is a text-only file that contains no special formatting. Most 
programs can read ASCII files. 



ASCII Files Used for Importing/Exporting 

When you export data from a meta-analysis, Meta-Stat creates two ASCII files. 
If you want to import data into Meta-Stat , you must create these ASCII files 
yourself. 

• A definitions file contains the variables for the meta-analysis. This file 
is named <meta-name>. ASD 

• A second file contains study data. This file is named <meta- 
m*roe>.ASM 

The <meta-name> is the name of the meta-analysis. 

Structure of the Definitions File 

The definitions file contains variables that are used in the meta-analysis. It 
has this filename: 

<meta-name>. ASD 

where <meta-name> is the name of the meta-analysis. 

Each record (line) in the file identifies a separate variable, and the fields in 
the record are delimited by commas. Alphanumeric fields are enclosed 
within quotes. 
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Field 


Contents 


1 


Numeric identifier of the variable 


2 


Eight-character name of the variable 


3 

6 


One-character identifier of the variable's type: 

B Blocking 
C Continuous 
D Discrete 
E Equation 


4 


Total length of the variable's data 


5 


Number of decimal places allowed for the variable's data 


6 


25-character description of the variable 

For blocking, continuous, and equation variables, a | 
separates the description from additional variable 
information. 



The following example shows the contents of a definitions file: 



1, "EFFECTSZ" , "C" , 8,3, "Effect Size" 

2, "UNBIASED", "E" , 8, 3, "Unbiased Effect Size | (1 - <3 / ( (4 * 

TOTAL_N) - 9))) * EFFECTSZ" 

3 , " EFFSR_GP" , "D" ,1,0, "Effect size source | l=Correlation ; 
2=T- Statistic ; 3=F-Statistic; 4=Chi-Square ; 5=Probability 
Level; 6=Standardized Difference; 7=Direct Entry; 8=Gain 
Scores (parametric) " 

4, "EFF_CLTR", "D", 1,0, "Effect size cluster" 

5, "PUB_YR» , "C" ,2,0, "Publication Year" 

6, "EXP_N" , "C» , 10, 0, "Experimental Group Size" 

7, "CTRL_N'\ "CM 0,0, "Control Group Size" 

8, "TOTAL_N","E", 10,0, "Total Sample Size | EXP_N + CTRL_N" 

9, "WEIGHT", "E" ,8,3, "Inverse of the variance | 

( 2 * T0TAL_N* EXP_N* CTRL_N ) / ( ( 2 * SQR ( TOTAL_N ) ) + 
(EXP_N»CTRL_N«SQR (UNBIASED) ) ) " 



BEST COPY AVAILABLE 



Chapter 4 



79 



67 






Managing Files 



Meta-Analvsis User’s Guide 



Structure of the Study-Data File 

The study-data file contains study data. It has this filename: 

<meta~name>. ASM 

where <meta-name> is the name of the meta-analysis, up to eight characters 
long. 

Each record (line) in the file identifies a separate study, and the fields in the 
record are separated by commas. Alphanumeric fields are enclosed within 
quotes. 

NOTE: Missing data — that is, a variable for which no value is 
available — are identified by -999999. 

The table below shows the structure of the study-data file: 



Field 


Contents 


1 


Eight-character name of the study 


2 


Authors of the study 


3 


Title of the study 


4-end 


Data for each variable, separated by commas. The record 
ends with a carriage return - line feed. 




Missing data are identified by -999999 
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The following example shows the contents of a study-data file: 
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Exporting a Meta- Analysis 

To export a meta-analysis to ASCII files, follow these steps: 

1 . Select File/Export ASCII 

Meta-Stat prompts you to identify the meta-analysis you want to export: 




2. Highlight the file you want to export, then press Enter. 

If necessary, first select the directory in which the file is located. For more 
information, see the section "Changing the Directory" earlier in this 
chapter. 

Meta-Stat creates two ASCII files that hold data from the meta-analysis. For 
more information about these files, see the section "ASCII Files Used for 
Importing/Exporting" earlier in this chapter. 
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Importing a Meta-Analysis 

To import data from ASCII files into Meta-Stat , follow these steps: 

1. Create the two ASCII files that will be imported. For more information about 
these files, see the section "ASCII Files Used for Importing/Exporting" 
earlier in this chapter. 

Warning: Use the extension ASD for the definitions file, ASM for the 
study-data file. For example: 

SAT. ASD 
SAT.ASM 

Make sure that you have not used the same name for an existing meta- 
analysis. For example, if you already have a meta-analysis called SAT, and 
you import the two files listed above, Meta-Stat will overwrite the existing 
meta-analysis. 

2. Start Meta-Stat. 
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3. Select Flle/lmport ASCII 

Meta-Stat prompts you to identify the definitions file. This is the file that 
contains variables: 
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4. Highlight the definitions file you want to import, then press Enter. 

If necessary, first select the directory in which the file is located. For more 
information, see the section " Changing the Directory " earlier in this 
chapter. 
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5. If your study-data file has a different name from the definitions file, Meta- 
Stat prompts you to identify the study-data file: 




6. Highlight the study-data file, then press Enter 

Meta-Stat imports the files and creates a meta-analysis. 
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Converting a Meta-Analysis for SPSS 

Meta-Stat can convert the data from your meta-analysis into two files for use 
with SPSS (Statistical Package for the Social Sciences). Follow these steps: 

1 . Change to the directory where you installed Meta-Stat. 

2. Type this command from the DOS prompt: 

M2SPSS <meta-analysis> 

where <meta-analysis> is the name of the meta-analysis you have created 
with Meta-Stat. 

3. Press Enter. 

Meta-Stat creates two output files: <meta-analy$i$>.INC contains variable 
definitions (referred to as an ’Include file' by SPSS), and <meta-analysis>.DAT 
contains study data. 

4. Check your screen for errors. The most common error occurs when the .INC 
file refers to a variable that is not defined until later in the file. Use your 
word processor to move the variable definition before the offending error. 

5. Use the two files as input to SPSS. Consult your SPSS documentation for 
more information. 

The .DAT file contains the data for all the variables in your meta-file, including 
the data derived from equations and blocking variables. If you are missing 
CTRL_N and EXP_N, for example, but have recorded TOTAL_N, M2SPSS will use 
your value in recording TOTAL_N, not the equation. Thus, M2SPSS lets you 
analyze the identical dataset. 

The functions used by meta-stat to provide the equation and blocking variable 
results are also provided in the include file. They are commented out with a 
leading '* ' so as not to override the recorded values. If you want to use the 
equations, use your word processor to remove the equations. 



74 



86 



Chapter 4 





Working with 
Variables 



Variables are characteristics of the studies in your meta- analysis. Meta-Stat comes 
precoded with several variables. You can add other variables specific to your meta- 
analysis. 

This chapter describes how to work with variables. 



Types of Variables 

You can use four types of variables: 



Type 


Use When ... 


Example 


Discrete 


The variable will 
have up to nine 
values, and you know 
the values in 
advance. 


A discrete variable called 
GENDER could have the values 
Male or Female. 


Continuous 


The variable can 
have any value. 


A continuous variable called 
HOURS could describe the 
number of hours that subjects 
received treatment, coaching, or 
other intervention. 
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Type 


Use When ... 


Example 


Blocking 


The variable will 
identify up to five 
groups of data, and 
the groupings are 
based on the values 
of other variables. 


A blocking variable could be 
used to group student grade 
levels. The variable could 
contain the values Elem, 
Middle, and High. 

Elem Grades 1-5 

Middle Grades 7-9 

High Grades 10-12 

The variable's equations could 
calculate the values based on a 
continuous variable that holds 
the actual grade level (1 to 12). 


Equation 


The value of the 
variable should be 
calculated by Meta- 
Stat, using mathe- 
matical operators and 
other variables. 


A precoded variable, 

UNBIASED, uses the sample size 
and Effect Size variables to 
remove Effect Size bias that is 
caused by a small study sample. 
See the next page for more 
information about this variable. 



Precoded Variables 

Meta-Stat comes precoded with several variables that are used for a typical 
meta-analysis. The table below describes these variables for effect-size meta- 
analysis. 



Variable 


Type 


Is Used to Identify ... 


EFFECTSZ 


Continuous 


Effect Size from a study, typically 
the difference between the means of 
two groups divided by a standard 
deviation. 






Meta-Stat can calculate the value of 
this variable for you. 


UNBIASED 


Equation 


Unbiased Effect Size. This 
variable's equation uses the sample 
size (TOTAL_N) and the Effect Size 
(EFFECTSZ) to remove Effect Size 
bias that is caused by a small study 
sample. The equation is: 






UNBIASED- (1 - (3 / ((4 * T0TAL_N) - 9))) * 
EFFECTSZ 
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Variable 


Type 


Is Used to Identify ... 


EFF$R_GP 


Discrete 


Source of the Effect Size value. 

The variable describes one of eight 
values. Meta-Stat fills in the value 
for you based on your entry for the 
Effect Size variable. 

Value Effect Size Source 

1 Correlation 

2 T-Statistic 

3 F-Statistic 

4 Chi-Square 

5 Probability 

6 Standardized 

7 Gain Score 

8 Direct Entry 


PUB_YR 


Continuous 


Year the study was published. 


EXP_N 


Continuous 


Size of the experimental group. 


CTRL_N 


Continuous 


Size of the control group. 


TOTAL.N 


Equation 


The total sample size. This equation 
uses the size of the experimental 
group (EXP_N) and the size of the 
control group (CTRL_N) to calculate 
the total sample size. The equation 
is: 

TOTAL.N - CTRL_N+EXP_N 


WEIGHT 


Equation 


The inverse of the variance. This 
equation uses the values of other 
variables to calculate a weighting 
value. 

The equation is: 

WEIGHT = (2 * TOTAL_N * EXP_N * CTRLN) / 
((2 * sqr(TOTAL_N)) + (EXP_N * CTRL_N * 
sqr(UNBlASED))) 


PUB__GP 


Discrete 


The type of publication in which the 
study appeared. 

The variable can hold one of five 
values: 

Value Tvne of Publication 

1 Journal 

2 Dissertation 

3 Paper 

4 Book 

5 Other 
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Creating New Variables 

You must create additional variables to describe the data from the studies in your 
meta-analysis. 

Initial Steps for All Types of Variables 

No matter which type of variable you want to create, the initial steps are the 
same: 

1 . If necessary, create or open a meta-file. See Chapter 4 for more information. 

2. Select Variable/New (*N) 

Meta-Stat displays the following screen: 
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3. Complete the type, ID, and Descr fields as shown below: 



Field 


Enter this ... 


Notes 


type 


A letter to identify the type 
of variable you are creating: 

Letter Variable Tvne 

B Blocking 

C Continuous 

D Discrete 

E Equation 


As soon as you identify 
the type of variable, 
Meta-Stat displays the 
appropriate screen for 
that type. The cursor 
moves to the ID field. 


ID 


An eight-character identifier 
that: 

• Begins with a letter 

• Contains no punc- 
tuation or special 
characters except the 
underline character (_). 


Meta-Stat uses the ID to 
identify the variable on 
selection screens. 
Therefore, use a 1 

meaningful identifier. 

Press Enter after you 
complete this field. 


Descr 


Up to 25 characters to 
identify the variable. 


Meta-Stat uses the 
description as a label for 
the variable on screens 
and reports. 

Press Enter after you 
complete this field. 
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4. Press F2 to see a list of the variables that you have already created for the 
meta-analysis. For example: 
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5. Press F10 when you have finished. 

Meta-Stat saves the new variable and returns you to the Variable menu. 
The sections below describe additional steps required for each type of variable. 
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Creating a Continuous Variable (Type C) 

When you create a continuous variable, you must identify the length of data you 
will input for the variable, including the number of decimal places you want to 
retain. 



Field 


Enter this ... 


Example 


length 


A number that identifies 
the maximum length of 
data you will enter for 
the variable, including 
the decimal point and 
decimal places. 


A length of 6 means that you 
will enter up to 6 characters 
total, including a decimal 
point if any. 


decimal 

places 


A number that identifies 
the number of decimal 
places you want Meta- 
Stat to retain. 

NOTE: Meta-Stat will 
round decimals, if 
necessary, when you 
perform data entry. 


A length of 6 with two 
decimal places means that 
you will enter up to 6 
characters total, including a 
decimal point and up to two 
decimal places — for 
example: 

512.18 

615.25 

If you enter three decimal 
places during data entry, 
Meta-Stat will preserve the 
significant digits internally, 
but will round the decimals to 
two places for display. For 
example, 52.896 will be 
rounded in displays to 52.90. 
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The following example shows a continuous variable called HOURS. The variable 
has a length of 4, with 2 decimal places: 
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Creating a Discrete Variable (Type D) 

Use a discrete variable to handle up to nine values that you know in advance. To 
assign a value to the variable during data entry, you will enter a number from 1 
to 9. Meta-Stat will then assign the corresponding label. 



Field 


Enter this ... 


Example 


Grp 


No entry required. The 
Grp value is the value you 
will enter during data 
entry. 


— 


labels 


The label that corresponds 
to the Grp value. These are 
optional. 


When creating a variable 
called Gender, you would use 
Female for the first label and 
Male for the second label. 

To assign a value during data 
entry, you would enter 1 to 
indicate female, 2 to indicate 
male. 
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The following example shows a discrete variable called PUB_TY. When entering 
data for this variable, you would enter 1 if the study was published in a journal, 2 
if it was published in a paper, and so on: 



7 • "'File _ Studies. f.' Wrj^lef ;^nali|sis- : djiarte; ( 7Cfttioi»s t _ Printer Info 

lilililr — — = — edit New WRiftBLE — — — — i ||i 

iiiliiiSl • : ■' ifliHi! 



I.! - .:. 

•; ! : : : : 



; • 1 ! • j j 

8 1 



“niidSSj’K 



iWM 

: 

j jl’jill t i I •;!') ! ‘t 

iiiiiiiiW 
1 Bill 1 ®' 

liShi! 

i : ji- ••• ■: 
:-i;i 

Jilliilllll 

£ tii!’ iiiliislii: 



Mo: 13 

type: J) 



ID: MBJHPr Descr : -?U?ilicAtf:«n ‘T^pe- 



Grp Labels 

1: Journal 
2: Paper - * 
3: Book 
i: Bissertatn 

5: Other -• 
6: 

7: 

8: 

9: 



label for data ualues 
-Fl-He lp F2HWBS F10-Cont inue ESC-Qu it- 



liislii i iiiiii in i i m iiiiiiiii mi iisiii aiiisi , n mini 

V^Fl-Help F10-flenu ' • . COftCH 





i . 






•i - 1 

i\ ' 

■•!■'■ ;■■■■■■■■• 

Pill ill 

■ : 

• ■ . 

..£ i 

• I: : 

iiiiii 

Itlli 

18 1 lljl i 
Hi 

mwim 

lliiilli 

13:38:23 



best COPY available 



84 



9 s 



Chapter 5 






Meta-Analysis User’s Guide 



Working with Variables 



Creating a Blocking Variable (Type B) 

Use a blocking variable when you want to arrange data into groups. 



Example of a Blocking Variable 

Suppose you have created a continuous variable that will describe the grade 
level (from 1 to 12) of study participants. You could create a second, 
blocking variable to assign a group: 



Grade Level 


Grade Group 


(Continuous) 


(Blocking) 


1 


Elem School 


2 




3 




4 




5 




6 




7 


Middle School 


8 




9 




10 


High School 


11 




12 





During data entry, you would enter the grade level of study participants. 
Meta-Stat would then use the blocking variable to assign a group. 



Advantages of Blocking Variables 

A blocking variable allows you to preserve your original data (the grade 
level, in the previous example). At the same time, the variable provides you 
with meaningful groupings of data that make for cleaner charts and displays. 

For example, the following chart was created using grade level as a discrete 
variable: 
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This chart, created with the blocking variable, might better suit your purposes: 
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Creating a Blocking Variable 

When you create a blocking variable, you must specify equations that Meta - 
Stat will use to group values. These equations refer to the values of other 
variables. 

The table below shows you how to create a blocking variable. 



Field 


Enter this ... 


Example 


Grp 


No entry required. 


— 


labels 


A label, up to 10 
characters long, to 
identify the group. This 
label will be displayed 
on screens and charts. 


A variable called Grade 
Group might use group 
labels such as Elem, 
Middle, and High. 


Conditional 

Equations 


The equation that 
Meta-Stat will use to 
group values. For more 
information, see the 
notes on equations 
below. 


The equations for the Grade 
Group variable described 
above would use the value 
of a second variable, Grade 
Level, to assign a value for 
Grade Group. 



When you create equations, be aware of the following: 

• You can use the following mathematical operators: 

<><=>=<> AND OR NOT ( ) 

• The equations must refer to variables that you have already 
created. Press F2 to display a list of the variables you have 
already created. You can then select a variable to be inserted 
into the equation. 

• The equation must result in a true or false condition. 
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The following example shows a blocking variable called GRD_GP. To group 
values, the equations for this variable use a continuous variable called 
GRDJ.VL. For example, if the GRD_LVL is between 1 and 6, Meta-Stat will 
assign a GRD_GP value of 1 (Elementary): 
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The following example shows a blocking variable called HOUR_GP. To group 
values, the equations for this variable use a continuous variable called HOURS. 
For example, if the value of HOURS is greater than 0 and less than 1, Meta-Stat 
will assign a H0UR_GRP value of 1 (short): 
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Creating an Equation Variable (Type E) 

Use an equation variable to calculate a value based on mathematical operators 
and other variables. 

When you create an equation variable, you must identify the length of data that 
will be displayed, including the number of decimal places you want to retain. 



Field 


Enter this ... 


Example 


length 


A number that identifies 
the maximum length of 
data that will be 
displayed, including the 
decimal point and 
decimal places. 


A length of 6 means a 
maximum length of 6 
characters, including a 
decimal point if any. 


decimal 

places 


A number that identifies 
the number of decimal 
places you want Meta- 
Stat to display. 

NOTE: Meta-Stat will 
preserve all significant 
digits but will round 
decimals, if necessary, 
for display. 


A length of 6 with two 
decimal places means a 
length of 6 characters total, 
including a decimal point 
and up to two decimal 
places — for example: 

512.18 

If the result of the calcula- 
tion contains three decimal 
places, Meta-Stat will pre- 
serve all three digits but 
will round the decimals to 
two places for display. For 
example, 52.896 will be 
rounded to 52.90. 


Equation 


The equation you want 
Meta-Stat to use. 


An equation to calculate the 
total sample size would add 
the size of the experimental 
group to the size of the 
control group. 
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When you create equations, be aware of the following: 

• You can use the following mathematical operators: 

+ -■*/() 

• The equations must refer to variables that you have already created. 
Press F2 to display a list of the variables you have already created. 

• The equation must result in a real number, so logs or square roots of a 
negative number are not allowed. 

• You can use the following mathematical functions: 



Function 


Description 


Example 


ABS 


Absolute value 


ABS(VARl) 


ATAN 


Arctangent, in radians 


ATAN(VARl) 


COS 


Cosine, in radians 


COS(VARl) 


EXP 


Exponential 


EXP(VARl) 


LN 


Natural logarithm 


LN(VAR3) 


ROUND 


Round result to whole 
number 


ROUND(TOT) 


SIN 


Sine, in radians 


SIN(VAR2 


SQRT 


Square root 


SQRT(TOT) 


TRUNC 


Truncate decimals (whole 
number without rounding) 


TRUNC(TOT) 
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The following example shows the equation for the Unbiased Effect Size: 
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Changing Your Variables 

You can change your variables in different ways. You can: 

• Edit a variable 

For example, you might want to change the description of a variable, or 
change the length of data that you can enter for the variable. 

• Copy a variable 

If two variables are similar, you might want to create the first variable, 
then copy it and edit the copy. 

• Delete a variable 
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If you no longer need a variable, you can delete it. 

The following sections describe how to change your variables. 

Editing a Variable 

You can edit a variable by selecting it from a list, or by highlighting the variable 
within a study. 



Editing by Selecting from a List 
1. Select Variable/Edit ( A D). 

Meta-Stat displays a list of your variables: 
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2. Move the cursor to the variable you want to edit, then press Enter. Meta - 
Stat displays a screen where you can edit the variable: 
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3. Press Enter or Tab to move the cursor to the field you want to edit. Then 
use the editing keys Del, Ins, and Backspace to edit the field. 

4. Press F10 when you have finished. 
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Editing by Highlighting a Variable Within a Study 

While editing a study, you can go immediately to the editing screen for a 
variable. 

1 . Highlight the variable by moving the cursor to it. 

In the following example, the highlighted variable is Variable 13, 

GRDJ.VL: 
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2. Press A D to display the editing screen: 
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3. Press Enter or Tab to move the cursor to the field you want to edit. Then use 
the editing keys Del, Ins, and Backspace to edit the field. 

4. Press F10 when you have finished. 
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2. Move the cursor to the variable you want to copy, then press Enter. 
Meta-Stat prompts you for the name of the new variable: 
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3. Use the editing keys Del, Ins, and Backspace to create the name of the new 
variable. 

4. Press Enter when you have finished. 
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Moving a Variable 

The order of variables determines the order in which you enter data for each 
study. If your data is set up in such a way that you want to enter one variable 
before another, you can change the order of the variables. 

Follow these steps to move a variable: 

1 . Select Variable/ Move 

Meta-Stat displays a list of your variables: 
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2. Move the cursor to the variable you want to move, then press Enter. 

Meta-Stat prompts you to identify the location of the variable you are 
moving: 




3. Move the cursor to the location where you want to move the variable. 

4. Press Enter when you have finished. 

Meta-Stat rearranges your variables and study data. 
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Deleting a Variable 

You can delete a variable that you no longer need. (However, you cannot delete 
a variable if other variables refer to it in equations.) 

Warning: Meta-Stat will remove the variable and all study data that you have 
entered for the variable. 

Follow these steps to delete a variable: 

1 . Select Variable/Delete 

Meta-Stat displays a list of your variables: 




2. Move the cursor to the variable you want to delete, then press Enter. 

Meta-Stat prompts you to confirm the deletion. Press Y or N. 

BEST COPY AVAILABLE 
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Working with Libraries 

Libraries are collections of pre-coded variables. Meta-Stat uses pre-coded 
variables, for example, when first defining an effect-size or correlation meta-file. 
You can create and use your own libraries of variables. You may, for example, 
want to import variables from a previous meta-analysis, or you may want to 
share variables and equations will colleagues. 

• To create a library 

1 . Select the variable you want to include in your library 

Select Save to Lib. Meta-Stat will then present a list of your current 
variables. Push Enter to select variables. A check will appear next to the 
selected variable. You may select one or all of the current variables to 
include in your library. If you push Enter again, the variable will be de- 
selected and the check mark will no longer appear. 

F9 to select all the variables on the list. 

Space to deselect all the variables on the list. 

F10 when you are done. 

2. Specify the library in which you want to save the variables. 

You can either add the variables to an existing library by selecting an 
existing library, or you can create a new library by pushing Alt-C and 
entering a new file name. Libraries should be saved with an .MLB 
extension. 

• To edit an existing library 

1 . Select delete Ubravy Variable. 

2. Select a Library. 

3 . Select the variable you want to delete by moving the cursor to the 
variable and pushing Enter. 

• To use variables from an existing library. 

1. Select Load from Lib. 

2. Select the desired library name. 

3. Select the variables you want to add to your current meta-file. The 
selection follows the same conventions as Save to Lib discussed above. 
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Study Data 



This chapter describes how to create studies, enter study data, and display and print the 
data in various ways. 



Creating a New Study 

You must create a study before you can enter data for it into Meta-Stat. Follow 
these steps: 

1 . If necessary, open the meta-file to which you want to add the study: 

• Use File/ New to create a new meta-file, or 

• Use Flle/Open to open an existing meta-file 

See Chapter 4 for more information about creating and opening meta-files. 
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2. Select Studies/New. 

Meta-Stat displays this screen: 




3. Fill out the following information for the study. Press Tab or Enter to move 
from one field to the next. 
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Field 


Enter this ... 


study id 


An identifier of the 
study. 

Limit: 8 characters 


publication 


Two digits to identify 


year 


the year in which the 
study was published. 


author(s) 


The authors of the 
study. 

Limit: 8 characters 


title 


The study title. 
Limit: 50 characters 



Meta-Stat uses the ID to 
identity the study on 
screens and reports. 
Therefore, use a descriptive 
name, such as the first 8 
characters of the author's 
name. The ID cannot start 
with a number or 
punctuation code. 



Enter the last names first to 
make your screens and 
reports more readable. 



The following example shows a completed screen: 
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4. Press F10 to save the new study. 

Entering Data for a Single Study 

After you create a study, you can enter study data for each of the variables in 
your meta-analysis. This section shows you how to enter data for a single study. 
You can also enter data for multiple studies, as described in the section 
"Entering Data for a Group of Variables" later in this chapter. 

To enter data, follow these steps: 

1. Select Studies/Edit. 

Meta-Stat displays this screen: 
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2. Highlight the study you want to edit, then press Enter. 

Meta-Stat displays an editing screen. The study identifiers are shown at the 
top of the screen, followed by the variables and study data: 



File Studies 


Uar table Analysis Charts Options Printer 


Info 


■STUDY 1 IB: CASTILLE 

\mm Nun: 6 of 31 
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AUTHORS; Cast i lie j H.R. 
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General Editing Tips 

To enter or edit data in a study, you can type directly at the cursor's location. In 
addition, you can use the following keys: 



Press this key 


To do this 


Enter 

or cursor-down 


Move the cursor to the next field for editing. 


Back-Tab 
or cursor-up 


Move the cursor to the previous field for 
editing. 


PgDn 


Display the next group of variables 


F10 


Save the study data and return to the Studies 
menu. 


F7 


Save the study data and display the previous 
study. 


F8 


Save the study data and display the next 
study. 



F10 is the important key to remember. Press this key to save the data you have 
entered. 
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Entering the Effect Size 

The first variable in effect-size meta-analysis is EFFECTSZ, the effect-size for a 
study. You can enter the Effect Size directly, by typing it directly into the 
EFFECTSZ field, or you can use an Effect Size calculator built into Meta-Stat. 

Follow these steps to enter the Effect Size using the effect size calculator. 

1 . With the cursor in the EFFECTSZ field, press Tab. 

Meta-Stat displays a screen on which you can select the source of the Effect 
Size: 
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2. Use the cursor keys to highlight the effect-size source for this study, then 
press Enter. 

Meta-Stat displays a screen where you can enter information about the 
source of the Effect Size. For example, if you want to calculate the Effect Size 
from the F Statistic reported in the study, Meta-Stat displays this screen: 
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3. For each element of the source statistic, press Enter to display a window 
where you can enter needed information. 

For example, with the cursor next to F Statistic, press Enter to display a 
window where you can enter the F Statistic: 




4. Use the same process to enter additional information. For example, referring 

to the F Statistic example shown above: 

• Press cursor-down to highlight Degrees of freedom for the denominator. 

(The degrees of freedom for the numerator is set to 1 as the conversion 
formula is only applicable when the F Statistic is calculated for two 
groups.) 

• Press Enter 

• Enter the denominator degrees of freedom in the window that Meta-Stat 
displays 

BEST copy available 
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• Press Enter to close the window 

• Direction refers to whether the mean differences show a positive result 
(i.e., treatment group is greater that the control group) or vice versa. 



Working with Equations and Blocking Variables 

You cannot directly enter data for variables that have been defined as equations 
or blocking variables. Meta-Stat calculates the values of these variables for you. 

However, you can enter data for an equation if a component of the equation is 
missing. For example, if a study did not use control subjects, you can skip the 
CTRL_N (control group size) variable. Meta-Stat then allows you to directly enter 
T0TAL_N (total sample size), because one of the components of the T0TAL_N 
equation is missing. 



Viewing Another Study During Data Entry 

While editing one study, it is often helpful to be able to view the contents of 
another study. 

1 . Select Studies/View Another 

2. In the list that Meta-Stat displays, select the study you want to view by 
highlighting it and pressing Enter. 
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Meta-Stat displays the second study below the first: 



f File Studies 
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Info 
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Entering Data for a Group of Variables 

As an alternative to entering data for all variables for a single study, you can 
enter data for a group of variables. You select the variables you want to work 
with, and Meta-Stat displays only those variables for each study. This allows for 
fast entry. 

Follow these steps: 

1 . Select Studies/Edit Grp of Vars 

Meta-Stat displays a list of variables. 

2. Use the cursor keys to highlight each variable you want to work with, then 
press Enter. 

In the following example, two variables have been selected: 
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3. Press F10 when you have finished selecting variables. Meta-Stat displays the 
variables for the first study in your meta-file: 



File Studies Variable Analysis Charts Options Printer Info 
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4. Use the normal editing features of Meta-Stat to enter or edit data for the 
variables. 

5. Press F8 and F7 to display the next and previous studies, respectively. 



Keeping Studies Sorted 

When you create a new study, Meta-Stat adds it as the last study in your meta- 
analysis. When you display a list of studies, or want to select a study to edit, 
Meta-Stat displays the studies in the order in which you created them. 
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To make it easier to find studies in your meta-analysis, you should periodically 
sort your studies. You can sort studies by: 

• Study ID 

If you used the author's last name as the variable ID, this sort order 
provides you with an alphabetical list of your studies. 

• Variable values 

For example, if you want to sort your studies chronologically, you could 
sort them according to the PUB_YR variable. You can sort in ascending or 
descending order. 

To change the sort order, or to keep studies sorted after you add new studies, 
follow these steps: 

1. Select Studies/Sort 

Meta-Stat asks you to select the type of sort you want to perform: 
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2. Highlight the type of sort you want to perform, then press Enter. 

If you are sorting by study IDs, Meta-Stat sorts your studies. If you are 
sorting by variable values, Meta-Stat asks you to select the variable to use: 
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3. Highlight the variable you want to use to sort the studies, then press Enter. 
Meta-Stat asks you to select a sort sequence: 




4. Highlight Ascending or Descending, then press Enter. 
Meta-Stat sorts your studies in the order you selected. 
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Displaying and Printing Study Data 

Meta-Stat provides a variety of ways to display and print data from your studies. 
(To find out about printing statistics and graphs, see the Chapters 6 and 7.) 

Displaying a List of Studies 

From within Meta-Stat , you can quickly display a list of studies. 

Use Studies/ view Reference to display the list: 
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Printing Studies 

You can print all information you have entered for a single study. Follow these 
steps: 

1. Use Printer/ Destination, if necessary, to select where you want to send the 
printed information. See Chapter 2 for more information about this menu 
option. 

2. Select File/Print Reports. 

Meta-Stat prompts you to select what you want to print: 




3. Select One Study to print a single study. Select Entire Database to print all 
studies in the meta-analysis. 
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If you selected One Study, Meta-Stat prompts you to select the study you 
want to print: 




4. If you are printing a single study, highlight the study you want to 
print and press Enter. 
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Meta-Stat prints the studies you selected: 



META 

LMP Associates 
Meta -file NAME 


PRINTED 

, Inc. 

: 


: 06/02/93 

12:36:06 

PAGE: 1 


STUDY ID: ALLEN TITLE: Effectiveness of study counseling 




STUDY NUM: 1 


AUTHOR (S) . Allen, G. J. 




VAR VAR 


VARIABLE 


DATA 


# ID 


DESCRIPTOR 


VALUE 


1 EPFECTSZ 


Effect Size 


0-918 


2 UNBIASED 


Unbiased Effect Size 






(1 - (3 / {(4 * TOTAL_N) - 9))) * EPFECTSZ 


0.909 


3 EFFSR_GP 


Effect size source | l=Correlation,- 2»T-Statistic; 
3-F-Statistic; 4-Chi-Square,- 5=Probability Level; 
6=Standardized Difference; 7oDirect Entry; 8=Gain 






Scores (parametric) 


2 


4 EFF_CLTR 


Effect size cluster 


0 


5 PUB_YR 


Publication Year 


67 


6 EXP_N 


Experimental Group Size 


41 


7 CTRLN 


Control Group Size 


40 


8 TOTAL_N 


Total Sample Size 






EXP_N + CTRL_N 


81 


9 WEIGHT 


Inverse of the variance 

(2*TOTAL_N*EXP_N*CTRL_N) / ( ( 2 * SQR ( TOTAL_N) ) + (EXP_N* 






CTRL_N*SQR (UNBIASED) ) ) 


18.351 


10 PUBTY_GP 


Publication Type | l=journal; 2=dissert; 3=paper; 






4-book; 5«other 


2 


11 HOURS 


Num of hrs of coaching 




12 HOUR_GRP 


Duration of program 

loshorti (H0URS>0) and (HOURS < 1) 

2 -average: (HOURS >= 1) and (HOURS <=1.5) 






3-long: HOURS > 1.5 




13 GRD_LVL 


Grade Level 1 thru 18 


14 


14 GRD_GRP 


Grade Group 

1-Elementary: (GRD_LVL >= 1) and (GRD_LVL <•> 6) 
2=High Sch: (GRD_LVL >- 7) and (GRD_LVL <- 12) 
3-College: ( GRD_LVL >= 13) and (GRD_LVL <- 18) 
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Viewing Data for Selected Variables 

If you want to review quickly only some variables for your studies, you can 
select the variables you want to see rather than viewing study data for all 
variables. Follow these steps: 

1 . Select Studies/View Variables. 

Meta-Stat asks you to identity the variables you want to view. 

2. Move the highlight to each variable you want to view and press Enter. If you 
select a variable by mistake, press Enter a second time to deselect it. 

In the following example, three variables have been selected: 




best COPY AVAILABLE 
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3. Press F10 to display the variables for each study: 




BEST copy AVWLAB* r 
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Printing Data for Selected Variables 

You can print study data for only selected variables. Follow these steps: 

1 . Use Printer/Destination, if necessary, to select where you want to send the 
printed information. See Chapter 2 for more information about this menu 
option. 

2. Select Rle/Print Reports. 

Meta-Stat prompts you to select what you want to print: 
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3. Select Data for a Variable. 

Meta-Stat prompts you to identify the variables you want to print: 




4. Highlight each variable you want to print and press Enter. 

If you select a variable by mistake, press Enter a second time to deselect it. 
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5. Press F10 to print the variables. In the following example, the EFFECTSZ 
variable has been printed: 



META 

LMP Associates. Inc. 
Meta -file NAME. COACHO 


PRINTED. 


06/02/93 

13.11.12 

PAGE : 1 


VARIABLE i 


EFFECTSZ Effect Size 


Variable Type. 


Continuous 


STUDY ID 


TITLE 


AUTHOR 




EFFECTSZ 


ALLEN 


Effectiveness of study counseling 


Allen, G- J. 




0.918 


BAJTEL1 


Test-viseness and systematic desens 


Bajtelsmit, J. 




2.500 


BAJTEL2 


Test-wiseness and systematic desenB 


Bajtelsmit, J. 




0.500 


BAJTEL3 


Test-wieenesB and systematic desenB 


Bajtelsmit, J. 




0.200 


CALLEN 


The effects of instruction and prac 


Callenbach, C. 




0.200 


CASTILLE 


The effects of commercial tutoring 


Castille , H.R. 




0.750 


COSTAR 


Scoring high in reading. The effec 


Costar, E. 




0.200 


CROZIER 


An evaluation of the effectiveness 


Crozier, P. W. 




1.000 


DILliAR 


Efficacy of test-viseness on test a 


Dillard, Warrio 




1.900 


EAKIN 


The effects of an instructional tea 


Eakins , Green, 




0.500 


EMERY 


Standard vs individual hierarchies 


Emery, J. R. & 




1.000 



Copying a Study 

You might want to copy a study to: 

• Place a copy of the study on diskette for safekeeping 

• Experiment with the study 

(You can experiment with the copy rather than the original) 

• Add a study to your meta-analysis that is similar to an existing study 
(You can copy the study and make changes to the copy) 

best copy AVAILABLE 
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To copy a study, follow these steps: 

1. Select Studies/ Copy 

Meta-Stat asks which study you want to copy: 



File Studies Variable Analysis Charts Options Printer Info 
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BAJTEL1 

BAJTEL2 

BAJTEL3 

CALLEN 

CASTILLE 

COSTAR 

CROZIER 

DILLAR 
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V^Fl-Help FlO-Menu 



COACH0 



2. Highlight the study you want to copy and press Enter. 
Meta-Stat asks for the new name of the study. 
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Use Ins, Del, and other editing keys to change the name of the copy. In the 
following example, a study named CASTILLE is being copied to a study named 



CASTILL2: 




4. Press Enter to copy the study. 
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Deleting a Study 

If you no longer need a study, you can delete it. Follow these steps: 
l. Select Studies/Delete. 

Meta-Stat asks you to identify the study: 



File Studies Uariable Analysis Charts Options Printer Info 
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COACH0 



18:52:49 



2. Highlight the study you want to delete and press Enter. 
Meta-Stat asks you to confirm. 

3 . Press Y to confirm the deletion or N to cancel. 
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Meta-Stat provides a wealth of statistical tools to help you examine your data. 



Overview 

The appropriate role for inferential statistics in meta-analysis is not merely 
unclear, it is seen quite differently by different methodologists. 

In 1981, in the first extended discussion of the topic, Glass, McGaw and Smith 
raised doubts about the applicability of inferential statistics for the meta-analysis 
problem. Inference at the level of persons within studies (of the type address by 
Rosenthal 1984) seemed quite unnecessary to them, since even a modest size 
synthesis will involve a few hundred persons (nested within studies) and lead to 
nearly automatic rejection of null hypotheses. Moreover the chances are remote 
that these persons or subjects within studies were drawn from defined 
populations with anything approaching probabilistic techniques; hence, 
probabilistic calculations advanced as if subjects had been randomly selected are 
dubious. 

At the level of "studies," the question of the appropriateness of inferential 
statistics can be asked again, and Glass et al. seem to answer in the negative. 
They pointed out that there are two instances in which common inferential 
methods are clearly appropriate: when a defined population has been randomly 
sampled and when subjects have been randomly assigned to conditions in a 
controlled experiment. In the latter case, Fisher showed how the permutation test 
can be used to make inferences to the universe of all possible permutations. But 
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this case in of little interest to meta-analysts who never assign units to 
treatments. Glass et al. claimed that the typical meta -analysis virtually never 
meets the condition of probabilistic sampling of a population 1 . They took the 
position that inferential statistics has little role to play in meta-analysis: "The 
probability conclusions of inferential statistics depend on something like 
probabilistic sampling, or else they make no sense" (p. 199). 

It is a common habit of thought to acknowledge that many data sets fail to meet 
probabilistic sampling conditions , but to argue that one might well treat the data 
in hand "as if' it were a random sample of some hypothetical population. Under 
this supposition, inferential techniques are applied and the results inspected. This 
circumlocution has neither logic nor common sense to support it. Indeed, it 
seems to be little short of a rationalization for performing statistics that one has 
gone to the trouble to learn whether they are appropriate or not. If the sample is 
fixed and the population is allowed to be hypothetical, then surely the data 
analyst will imagine a population that resembles the sample of data. Hence all of 
these "hypothetical populations" will be reflections of the samples and there will 
be no need for inferential statistics. The researcher runs the risk of generalizing 
to what well may be a fictitious, and hence irrelevant, population. 

Hedges and Olkin (1985) developed inferential techniques that ignored the pro 
forma testing of null hypotheses and focused on the estimation of regression 
functions that estimate effects at different levels of study characteristics. They 
worried about both sources of statistical instability: that arising from persons 
within studies and that which arises from variation between studies. As they 
properly pointed out, the study based on 500 persons deserves greater weight 
than the study based on 5 persons in determining the response of the treatment 
condition to changes in study conditions. The techniques they present are based 
on traditional assumptions of random sampling and independence. It is, of 
course, unclear precisely how the validity of their methods axe compromised by 
failure to achieve probabilistic sampling of persons and studies. 

Rubin (1990) addressed most of these issues squarely and staked out a position 
that appeals to the authors of this manual: "...consider the idea that sampling and 
representativeness of the studies in a meta-analysis are important. I will claim 
that this is nonsense- we don’t have to worry about representing a population but 
rather about other far more important things" (p. 155). These more important 
things to Rubin are the estimation of treatment effects under a set of standard or 
ideal study conditions. This process, as he outlined it, involves the fitting of 



1 In one instance known to Smith, Glass & Miller (1980), the available population of 
drug treatment experiments was so large that it was in fact randomly sampled for the 
meta-analysis. 
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response surfaces (a form of quantitative model building) between study effects 
(Y) and study conditions (X, W, Z etc.). 

Where theorists disagree, technicians are well advised to leave open as many 
options as possible. Consequently, we have provided many methods of data 
analysis in Meta-Stat that address inferential issues, including Hedges's 
homogeneity tests, Rosenthal and Rubin's aggregate significance levels as well 
as conventional methods of analysis like regression, and a newly rediscovered 
approximate randomization test 2 . 

Where conditions of random sampling are met , the inferential techniques 
in Meta-Stat will have clear and undisputed application . In all other 
instances, it will be up to the meta-analyst to decide whether to apply them 
and how to interpret them. We provide fair warning that the inferential 
statistical techniques in Meta-Stat are easy to misapply and misuse. 



Common Functions 

Some functions are commonly available in the Meta-Stat statistical analyses. 
The following sections describe these functions. 

You can also export data to analyzed with SPSS. See Chapter 4 for more 
information. 



Selecting Variables to be Analyzed 

After you select the statistical analysis you want to perform, Meta-Stat displays 
a selection window. This window contains fields where you identify the 
variables for the analysis. 



2 We do not provide ANOVA as we could not identify any algorithms that are 
appropriate for use with effect-sizes. For Plain Statistics, you can use dummy coding with the 
REGRESSION routine to perform ANOVA, ANCOVA, and t-tests. If your criterion variable is 
UNBIASED, the group MEANS routine will provide Hedges's homogeneity tests — tests which 
are directly analogous to ANOVA. If your criterion variable is UNBIASED and you select 
WEIGHT as a weighting variable, the REGRESSION routine will give you Hedges's Q E and Q B 
and the corrected standard errors for the regression weights. 
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Selecting a Single Variable 

For many analyses, you can select only a single variable in particular 
fields: 

1. Highlight the desired field and press Enter. 

Meta-Stat displays a window that lists the variables that are 
appropriate for that field. 

In the following example from the Regression analysis, Criterion 
Variable was highlighted when Enter was pressed. In the window, 
Meta-Stat displays a list of variables: 



File Studies Variable Analysis Charts Options Printer Info 
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Regression Analysis - 

pEfoependent Uar i ab 1 e : EFFECTSZ 

Independent Variables: 

Dumny Uariables: 






Weight: 
Missing: 
Select If: 



Listuise 
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2. To select a variable, highlight the variable in the list by moving the 
mouse or using 1 . Then press Enter. 



Meta-Stat returns you to the selection screen and fills in the field 
with the name of the variable you selected. 
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Selecting Multiple Variables 

For some fields, you can select more than one variable. For example, 
you can identify more than one predictor variable for the Regression 
analysis. Follow these steps: 

1 . In the list of variables, move the pointer to the first variable you 
want to use and press Enter. 

Meta-Stat marks the variable with a check-mark. 

2. Repeat step 1 until you have marked all the fields you want to use: 
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3. Press F10 when you have finished. Meta-Stat returns you to the 
selection screen and fills in the field with the variable names. 
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Deselecting a Variable 

Meta-Stat remembers the last variable(s) you selected. To deselect one or more 
variables, call up the list of variables and press Esc. 

in the following example, the variable WEIGHT is currently selected as the 
weighting variable: 
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To deselect this variable: 

1 . Highlight the Weight field and press Enter. 



Meta-Stat displays a list of variables. The variable that is currently selected 
is highlighted. 
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2. Press Esc to deselect the variable. 



Weighting Your Data 

You can weight your data using any variable. Meta-Stat includes a predefined 
variable called WEIGHT that you can use for this purpose. WEIGHT is an equation 
variable that equals the inverse of the variance. See Chapter 5 for more 
information about this variable. 
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To select the weighting variable, move the pointer to the Weight field and press 
Enter. Then select the variable you want to use: 



File Studies Uariable Analysis Charts Options Printer Info 
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Selecting Data with a SELECT IF Statement 

You can use a SELECT IF statement to limit the analysis to data meeting certain 
criteria. Meta-Stat will select for the analysis only the studies for which the IF 
statement is true. 

For example, suppose you use a variable QUALITY to measure the quality of 
studies; a low-quality study is given a QUALITY value of 1 . When you perform an 
analysis, you can use SELECT IF to trim the low-quality studies from the analysis 
by selecting only studies for which QUALITY is greater than 1. 

1 . Move the pointer to Select If and press Enter. 



138 



ISO 



Chapter 7 











Meta-Analysis User's Guide 



Applying Statistical Techniques 



Meta-Stat displays a window where you can enter selection criteria. 




2. Press F2 to display a list of variables. 

3. In the list of variables, highlight the variable you want to use as part of the 
SELECT IF statement. Then press Enter. 
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Meta-Stat inserts the variable name in the selection criteria window: 



e Studies Variable Analysis Charts Options Printer 



Crosstabu 1 at ions 

Grouping Variable 1: PUBT¥_GP 

Grouping Variable 2: HOUR_GFP 

Weight: 

^Select If: 

- Fl-Help F2-Select ESC-Quit F10-Cont 




eaue blank to acce 



m 1 - Fl-Help, F2-Ust Variables 



ijjjj 


> t o 1 1.' 1 i ‘i 1’’ * ' t ! i i s 1 

i ' : > : . . 


. ’ii 1 ' ' “ii, i i 1 ii Hi** i 


• , ... . , ! 


l »! i; \ M ;;i isi: = i'!. : “'v ! x % i 

• ,,[ 1 1 : > 1 1 i i • 1 , 1 1 i • i,i i ! ),: < • j | ■ | • ■ • • > j • 1 • • ji *j{ • ; | > • v • • 


Vji4ieip MO-tlenu 






COACH 10:15:58 j 




4. Complete the IF statement by using parentheses, logical operators, and/or 
comparison operators. (Press FI to get more information about constructing 
an IF statement) 
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In the following example, studies will be selected if the value of QUALITY is 
greater than 1 and if the value of EFFECTSZ is greater than 0.5: 




BEST COPY AVAILABLE 
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Dealing with Missing Data 

For the Regression and Correlation analyses, you can use the Missing Data field 
to tell Meta-Stat how to handle studies for which data are missing. 

You can use two methods to handle missing data: 



Method 


Description 


Listwise deletion 


If a study does not have data for one of the 
variables, Meta-Stat ignores all other 
variables for that study. 

This is the default method. 


Pairwise deletion 


If a study does not have data for one of the 
variables, Meta-Stat still uses data for other 
variables in the study. 

Caution: Be careful when performing 
regressions without complete data. It is 
generally valid to do this only when you can 
assume that the data are missing randomly 
throughout the studies. 



To identify the method you want to use, move the cursor to the Missing Data field 
and press Enter. Then select the method you want to use. 



Using Dummy Variables 

The Regression and Correlation analyses allow you to treat discrete and blocking 
data as dummy variables. For example, consider the discrete variable PUBTY_GP, 
which comes precoded with Meta-Stat. When you enter data for a study, you use 
PUBTY_GP to identify where the study was published — in a Journal, Dissertation, 
Paper, Book, or Other place. 

During a Regression or Correlation analysis, Meta-Stat can treat these five 
categories as separate variables by assigning scores of 1 and 0 to them. For 
example, a study that was published in a Journal receives a score of 1 for the 
dummy variable Journal; it receives scores of 0 for the dummy variables 
Dissertation, Paper, Book, and Other. 
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Dummy variables take their name because they are created only as a 
convenience, so that they can be inserted into equations that are necessary for the 
analysis. 

Use the field Dummy Variables to identify the discrete or blocking data from 
which you want to create dummy variables. The following example uses 

PUBTY_GP: 



File 
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The following example shows a portion of Regression results for which dummy 
variables were created from PUBTY_GP: 



f METfi-STAT 06/25/93^ 

l REGRESS IOM . 



Variable 




Beta 


B 


St Err 


t 


P 


PUBTVJ3P 


Journal 


0.000 


0.000 


0.Z30 


0 .000 


— 


PUBT¥_GP 


dissert 


0.204 


0.346 


0.217 


1.597 


0.120 


PUBT7_GP 


paper 


0 .020 


0.047 


0.217 


0.216 


0.025 


PUBTV_GP 


other 


0.615 


1.664 


0.454 


3.663 


0.002 


Constant 






0.536 


0.124 


4.334 


0.000 



\jCtrl-P Print FIB Menu PgUp, PgDn MoCQflCH 11:57:53 1 J 



NOTE: To avoid the multicolinearity problem in computing regression weight 

standard errors the first group is dropped. Usually, this is the group 
with missing data. 
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Displaying Descriptive Statistics 

When examining your meta-analysis data, one of your first tasks will be to 
explore the distribution of many of your variables. 

The Descriptive Statistics analysis provides a basic description of a variable. The 
mean shows you the central tendency for a variable, while the standard deviation 
and variance describe the dispersion of data around the mean. These same 
statistics are calculated for the Unbiased Effect Size variable. 

Follow these steps to perform the analysis: 

1 . Select Analysis/ Descriptive. 

Meta-Stat displays a screen where you can identify the variable you want to 
analyze, 

2. Use Criterion Variable to identify the variable for which you want to display 
the statistics. 

3 . If necessary, identify a weighting variable and construct a SELECT IF 
statement. For more information, see "Common Functions" earlier in this 
chapter. 
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In the following example, Meta-Stat will display descriptive statistics for the 

Unbiased Effect Size variable (UNBIASED), using the WEIGHT variable for 
weighting: 




4. Press F10 to display the results. 
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Results Screen 



META-STAT 



09/10/93 



DESCRIPT IUE STATISTICS 



Criterion Uariable: 
Weighting Uariable: 
Select If: 



UNBIASED 

WEIGHT 



Unbiased Effect Size 
Inverse of the variance ™ 



Number of Ualid Observations: 
Number of Missing Observations: 



29 

1 



O 





unweighted 


weightc 


Ck Mean 


0.591 


0.891 


V Median 


0.479 


0.479 


Uariance 


0.376 


0.428 


Stand. Dev. 


0.613 


0.654 


Min Ualue 


-€.287 


-0.287 


Max Ualue 


2.087 


2.087 


Range 


2.374 


2.374 


® Hedges' 8 Ht 


116.965 Cp 


= 0.090) 



Ctrl-P Print F10 Menu T1 next variable PgDn more 



22:26:48 



This results screen: 

O Identifies the variables you selected for the analysis and the SELECT IF 
statement you used to select data. The criterion variable is the criterion 
variable you selected. 

© Lists the number of studies for which data were available for the criterion 
dependent) variable. 

© Provides weighted and unweighted statistics for the criterion variable. 
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Value 


Description 


Mean 


Mean value, equal to the sum of the values 
divided by the number of cases (studies). 
The mean is the most common measure of 
central tendency. 


Median 


Median value, which is either the value of 
the case that lies exactly on the 50th 
percentile, once the cases have been 
ordered by rank from highest to lowest, or 
the value that is half way between the two 
inner most cases. 


Variance 


The dispersion of values around the mean, 
roughly equal to the average squared 
deviation from the mean. The smaller the 
variance, the more homogeneous the data. 
This is the variance for the presented 
dataset, i.e., N is used as the denominator . 


Stand. Dev. 


Standard deviation, equal to the positive 
square root of the variance 3 


Min Value 


Minimum value of the variable 


Max Value 


Maximum value of the variable 


Range 


The range between the minimum and 
maximum values 



© If you used the Unbiased Effect Size as the criterion variable, Meta-Stai 
shows the results of Hedges's Homogeneity Test. This tests the hypothesis 
that the Effect Sizes for the studies are from different populations — that is, 
that they are not homogeneous. If the statistic is large (or equivalently if 
the value of p is low, say less than .05), Hedges's test indicates that there is 
a low probability that the Effect Sizes are from the same population. You 
can conclude that the effect sizes are homogeneous, i.e. the model of a 
single Effect Size is consistent with the data. 



3 To convert from the variance of a population to the unbiased 
population estimate based on a sample, multiply by n/(n-l). 
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Second Results Screen for Unbiased Effect Size 



If you used the Unbiased Effect Size as the criterion variable, Meta-Stat shows a 
second screen of descriptive statistics. You can view this screen by pressing 
PgDn on the first screen. 



META-STAT 



09/10/93 



DESCRIPT IUE STATISTICS 



Criterion Uariable: 
Weighting Uariable: 
Select If: 



UNBIASED 

WEIGHT 



Number of Ualid Observations : 
Number of Missing Observations: 



Unbiased Effect Size 
Inverse of the uariance 



29 

1 



unweighted 



we ighted 



Mean ES 
Fisher's Z 
Equiu ES 
Equiu R 
Equiu R SQ 
Failsafe N 
Uar Ratio 



0.591 

1.497 

0.579 

0.278 

0.077 

0 

3.750 



Cp=8 .13) 



0.891 

6.661 

0.912 

0.415 

0.172 

326 

56.482 



Cp=O.00) 



Ctrl-P Print F10 Menu fl next uariable PgUp basic 



22:27:17 






J 



O 



Some information is repeated from the first results screen. 



© 



The table presents the best estimate of the Effect Size for the population of 
studies. 
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Value 


Description 


Mean ES 


Mean Effect Size 


Fisher Z 


A transformation that normalizes the Effect 
Size and provides a statistic that allows one 
to calculate the probability (p) of results 
under a null hypothesis of that the 
population Effect Size is zero. 


Equlv ES 


The equivalent Effect Size for the Fisher Z. 
This is the best estimate of the true Effect 
Size for this population of studies. 


Equlv R 


A correlation that is equivalent to Equlv ES. 


Equlv R SQ 


The amount of variance accounted for in 

the Effect Size. 


Failsafe N 


The number of studies with contradictory 
results that would have to be collected in 
order to reduce the probability of the 
combined findings to statistical 
insignificance atp=.05 (Rosenthal, 1990). 


Variance Ratio 


The ratio of the error variance to the 
variance of the Effect Sizes. This provides 
a homogeneity measure. Schmidt and 
Hunter (1990, pages 281-338) state that you 
can conclude that the data are 
homogeneous, i.e., all the variance is due to 
forms of error, if the ratio is .75 or larger. 



Performing a Regression Analysis 

Use a regression analysis to investigate the relationship between predictor 
variables and a dependent or criterion variable, which is typically the Effect Size. 

For example, in a meta-analysis of the effects of coaching on SAT scores, we 
might be interested in predicting the effects of coaching from predictor variables 
such as the number of hours of coaching and the grade level of the students. The 
regression analysis provides an equation that identifies how the values for hours 
and grade level must be weighted and summed to provide the best prediction of 
the coaching effects. The analysis also shows the accuracy of the equation and 
indicates how much of the variation in coaching effects is caused by hours and 
grade level. 
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To perform a regression analysis, follow these steps: 

1 . Select Analysis/Regression. 

Meta-Stat displays a screen where you can identify the dependent and 
predictor variables. 

2. Use Dummy Variables to treat each category within a blocking or discrete 
variable as a dummy variable. For more information, see the section 
“Using Dummy Variables" earlier in this chapter. 

3. If necessary, identify a weighting variable, a method to handle missing 
data, and construct a SELECT IF statement. For more information, see 
“Common Functions" earlier in this chapter. 

In the following example, a regression analysis will be performed to 
predict Unbiased Effect Size based on Hours and Grade Level: 



File Studies Uariable Analysis Charts Options Printer Info 

•••'•T' H'iiii’.liiji.im 1 \\V 5 1 j i r<< II «- \ 1 , ,, , it , ,!,;!!!!!"! •"» ’ i > m Hi< '■ 
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' ' - 1 • Ill 1 1 , ' , . I - , ' 



Regression Analysis 



Dependent Uariable: UNBIASED 

Independent Variables: HOURS GRD_LUL 

ummy Variables: 



Weight: 
Hissing: 
Select If : 



List wise 



1 — Fl-Help F2-Select ESC-Quit FIB-Cont. — l j| 



^Fi^Help FlB^ieim 




COACH 



9:30:21 



4. Press F10 to display the first screen of the analysis. 
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4. Press F10 to display the first screen of the analysis. 



First Results Screen: Summary Information 



META-STAT 



09/16/93 



REGRESSION 



Criterion Uariable: 
Predictor Uariables: 
Listwise deletion 
Solution took 2 iterations 



UNBIASED Unbiased Effect Size 
HDURS GRDJ.UL 



A Multiple R 
^ Adjusted R-Square 


0.8791 

0.7538 


R Square 
Standard Error 


0.7727 

0.3176 




DF 


Sun of 
squares 


Mean 

square 


F P 


A Regression 
Residual 


2 

24 


8.2299 

2.4204 


4.1150 

0.1008 


46.803 0.0000 

o 


Ctrl-P Print F10 Menu 






PgDn More 


22:29:21 



This screen: 

O describes the variables you selected for the analysis and the method that 
Meta-Stat used to handle missing data. 

© describes the strength of the relationship between the predictor variable(s) 

and the criterion variable. 



152 



161 



Chapter 7 






Meta-Analysis User’s Guide 



Applying Statistical Techniques 



Value 


Description 


Multiple R 


Correlation between predictors and the 
criterion variable. The magnitude R is an 
index of the strength of the relationship. 


R Square 


The square of R, indicating the amount of 
variance accounted for by the predictor 
variables. In the example above, 77 percent 
of the variation in Unbiased Effect Size is 
explained by the predictor variables. 


Adjusted 
R Square 


The adjusted R Square value. 


Standard Error 


The standard deviation of the differences 
between the actual and predicted values of 
the criterion variable (i.e., the standard 
deviation of the residuals). 



The summary table describes actual variances (mean squares) of the 
predicted values and the residual values. 

© The p value is the probability that the observed degree of relationship, or 
one stronger, could have been obtained by sampling error from a 
population in which the multiple R is 0. 
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Second Results Screen: 

Coefficients and Standard Errors 

To view the second screen of regression results, press PgDn on the first screen. 




154 



166 



Chapter 7 






Meta-Analysis User's Guide 



Applying Statistical Techniques 



o 



Shows the following information for each prediction equation: 



Value 


Description 


Constant 


The predicted value of the criterion variable 
when the predictor variables are 0. In the 
example above, the predicted value for Unbiased 
Effect Size is -0.388 when the HOURS and 
GRD_LVL variables are 0. 


Beta 


The beta weight, or the standardized regression 
coefficient: 




This value is calculated after the dependent and 
predictor variables have been standardized to 
have unit variance (that is, a standard deviation 
of 1 for all variables). The standardized 
coefficients are useful when the predictor 
variables are measured in different units, such as 
a duration in hours and a study date in years. 
Using these coefficients allows one to compare 
the relative influence of each predictor variable 
(see Darlington, 1968). In the example above, the 
predicted Unbiased Effect Size increases by 
0.168 standard deviation units for every increase 
of one standard deviation unit in the HOURS 
variable. The predicted Unbiased Effect Size 
increases by 0.745 standard deviation units for 
every increase of one standard deviation unit in 
the GRD_LVL variable. 


B 


The nonstandardized regression coefficient. 




This value indicates the expected change in the 
criterion variable with a change of one unit in the 
associated predictor. In the example above, the 
predicted Unbiased Effect Size increases by 
0.191 for every unit increase in the HOURS 
variable, while the predicted Unbiased Effect 
Size increases by 0.095 for every unit increase in 
the GRD__LVL variable. 


St Err 


The standard error of each non-standardized 
regression coefficient. This value indicates the 
variability that would arise in B from repeated 
sampling and estimation of B with samples that 
are the same size as this one. 
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The p value shows the probability of obtaining a beta this size or more 
discrepant from zero in sampling from a population in which the relevant 
beta is zero. 



Third Results Screen: Correlations 

To view the third screen of regression results, press PgDn on the second screen. 




o 



Provides a way to evaluate the contributions of each of multiple predictor 
variable to the variation in the criterion variable. 
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Value 


Description 


R 


The correlation between the criterion and 
predictor variables. Larger numbers, 
approaching +1.0 or -1.0 indicate a stronger 
correlation. The sign of R indicates a 
positive or negative relationship. 


Mean 


The average value of the variable. 


St Dev 


The standard deviation for the variable. 
This is the square root of the unbiased 
population variance estimator (N- 1 is the 
denominator). 


N 


The number of studies for which data were 
available. 



Chapter 7 



169 



157 





Applying Statistical Techniques 



Meta-Analysis User’s Guide 



Fourth Results Screen: Residuals 



To display the fourth results screen, press PgDn. Meta-Stat asks if you want to 
record predicted values. If you answer Y,for "Yes", Meta-Stat will save the 
predicted values for the criterion variable in a new variable called Y_HAT. For 
more information, see the section below called "Recording Predicted Values." 



META-STAT 



09/10/93 



REGRESS I DM 



Study ID 




0 v . 


® Residual 


ALLEM 

BAJTEL1 


0.9093 

Missing 


Missing 

1.6089 


— 


BAJTEL2 

BAJTEL3 


0.4901 

0.1915 


0.3728 

Missing 


0.1172 


CALLER 


0.1946 


Q.1829 


0.0117 


COSTAR 


0.1931 


0.3161 


-0.1230 


CROZIER 


0.9577 


0.0880 


0.8698 


DILLAR 


1.8817 


1.7033 


0.1784 


EAR IN 


0.4921 


0.7532 


-0.2611 


EMERV 


0.9897 


0.9616 


0.0281 


GARLT 


-0.2873 


-0.0076 


-0.2798 


GRDSS 


0.8889 


0.9043 


-0.0154 


JACOBS 


1.3893 


1.1897 


0.1995 


JOHNS 


0.9909 


1.0577 


-0.0668 


J0MGS1 


0.2873 


0.4678 


-0.1804 


J0MGS2 


0.3831 


0.2588 


0.1243 



O 

Residual Plot 

I - i -II 1--1 



- t 



Ictrl-P Print F10 Menu 



PgUp, PgDn More 22:39:67 J 



O Y is the actual value of the criterion variable. 

0 Y' is the estimated or predicted value for the criterion variable, based on the 
predictors. This predicted value is saved in the new variable Y_HAT, if you 
told Meta-Stat to save the values. 

0 Residual is the difference between Y and Y'. This is the error in prediction. 
The closer the values are to the vertical axis, the smaller the error. 

0 Meta-Stat shows a plot of the residual values. 
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Recording Predicted Values 

On the fourth results screen for the regression analysis, Meta-Stat asks: 

Do you want to record predicted values? 

If you answer Y, for "Yes", Meta-Stat will save the predicted values for the 
criterion variable — typically the Effect Size — in a new variable called Y_HAT. 
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You may then want to create a second new variable to hold the residual 
amount — that is, the difference between the predicted value for Effect Size 
(Y_HAT) and the actual value. 
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Plots of the residuals can help you assess whether your data meets the 
assumptions required of least squares regression analysis. We recommend the 
following plots (after Draper and Smith, 1966): 

1 . Bar Graph of the frequencies of the residuals values. If the model is 
correct, the plot should resemble observations drawn from a normal 
distribution with a zero mean. (Since an intercept is fit, the mean must be 
zero; but are the residuals normally distributed?) 

2 . Scatterplot of the residuals against Y_HAT. 

3. Scatterplots of the residuals against each of the predictor variables. 

If your plots show outliers or indicate possible curvilinear relationships, you may 
want to try one or more of the following: 

a) Checking your data entry and evaluating the veracity of some data 
points. 

b) Constructing a SELECT IF to trim suspicious datapoints. 

c) Adding or removing predictor variables. 

d) Transforming the criterion variable. 

e) Transforming one or more predictor variables. 

f) Adding a weighting variable. 

g) Using Non-linear regression. 
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Using Dummy Variables to 
Explore Differences Between Groups 

If you tell Meta-Stat to create dummy variables from a discrete or blocking 
variable, you can explore the differences between the groups identified by that 
variable. 

The following example shows regression results for which dummy variables 
were created from the variable PUBTY_GP. The results permit the comparison of 
studies based on the place where the studies were published: 



f HETfi-STAT 06/25/93^ 

i REGRESSION . 



Variable 




Beta 


B 


St Err 


t 


P 


FUBTY_GP 


Journal 


0.000 


0.000 


0.230 


0.000 





FUBTY_GP 


dissert 


0.204 


0.346 


0.217 


1.597 


0.120 


PUBTY_GP 


paper 


0.020 


0.047 


0.217 


0.216 


0.625 


PUBTYJ3P 


other 


0.615 


1.664 


0.454 


3.663 


0.002 


Constant 






0.536 


0.124 


4.334 


0.000 



y^Ctrl-P Print F10 Menu PgUp, PgDn NoCQACH 11:57:53 J 



The summary table obtained when using Dummy coding is the same as the 
summary table that would be obtained from an analysis of variance or 
covariance. For more information about dummy variables, see the section "Using 
Dummy Variables" earlier in this chapter. 

NOTE: If the criterion variable is UNBIASED and the weighting variable is 

WEIGHT, Meta-Stat follows the regression procedures for effect-sizes 
outlined in Hedges and OUdn (1985, 167-188). The sums of squares 
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are reported as homogeneity statistics and are individually tested. 
The standard errors for the regression weights are corrected by 
dividing by the positive square root of the mean square error. 



Testing Mean Differences 
Between Groups with Randomization 

Using the Randomization analysis, you can test the mean differences between 
groups that are represented by discrete or blocking variables. Randomization is a 
useful tool because it makes no assumptions about the distribution of data, such 
as the normality assumption. Instead, the actual group of studies in hand is used 
for the analysis. 

During the analysis, Meta-Stat calculates the mean value of the criterion variable 
for each group, then calculates the difference in means between the groups. Then 
Meta-Stat conducts up to 5,000 randomizing iterations to determine whether this 
actual difference is in fact unusual. During each iteration, Meta-Stat randomly 
draws groups from your group from the aggregate group of studies and calculates 
the differences in mean values for the groups. 

After completing all iterations, Meta-Stat displays the probability of getting a 
difference in means that is equal to or larger than the actual difference. If the 
probability is high, you can assume that the actual difference in means is not 
unusual, because there is a high probability of finding this difference or one 
larger using randomly partitioned data. But if the probability is low, you can 
assume that the single obtained difference is unusual. 

To use the Randomization analysis, follow these steps: 

1. Select Analysis/ Randomization. 

Meta-Stat displays a screen where you can identify the dependent and 
predictor variables. 

2. Use Criterion Variable to identify the variable that you want to measure for 
the groups. 

3. Use Grouping Variable to identify the discrete or blocking variable for 
which you want to test differences between groups. 

4. Optionally, use Blocking Variable to control for the effects of another 
variable without showing the variable on the results screen. 
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5. Select the Number of Iterations you want to use. Use fewer iterations when 
you are exploring. Use more iterations to get more precise results. 

Note that your results will be slightly different each time you perform the 
analysis, due to differences caused by randomization. 

6. If necessary, identify a weighting variable, a method to handle missing 
data, and a SELECT IF statement. For more information, see "Common 
Functions" earlier in this chapter. 

In the following example, Meta-Stat uses 500 iterations to measure 
differences in EFFECTSZ between groups identified by the H0UR_GRP 
variable. In other words, the analysis will test whether the variable 
H0UR_GRP has some relationship to the value of EFFECTSZ. 
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7. Press F10 to display the results of the analysis. 



META-STAT 



08/10/93 



APPROXIMATE RANDOMIZATION TEST OF MEAM DIFFERENCES 



Criterion Uariable 
Group ing Uariable 
30 Iterations 



UNBIASED Unbiased Effect Size 
PUBVRGRP Year of publication 



Mean Differences and (probabilities) 
Means and sample size on diagonal 





60-70 


76-80 


71-75 


81+ 


60-79 


1.1459 


0.5885 


0.6136 


1.0489 




C5) 


(0.129) 


(0.129) 


(0.032) 


76-80 




0.5574 


0.0251 


0.4605 






C15) 


(0.968) 


(0.226) 


71-75 






0.5322 


0.4353 








(5) 


(0.258) 


81+ 








0.0969 










(4) 



Ctrl-P Print FiO Menu 



22:39:59 



During the analysis, Meta-Stat first calculated the mean Effect Size that was 
obtained for each group (as represented by the H0UR_GRP variable). Meta-Stat 
then calculated the difference in obtained means between each group. 

Next, for each iteration of the analysis, Meta-Stat randomly selected studies 
from the entire collect of studies (28 in this example) to create three groups: one 
with 7 studies, a second with 16 studies, and a third with 5 studies. (Using this 
random partition ensures that, for the subsequent calculations, there is no 
difference between the groups.) For each group, Meta-Stat calculated the 
average Effect Size; it then calculated the differences between these averages. 
Meta-Stat then noted whether the difference was larger than the obtained 
difference between the actual groups. After the last iteration, Meta-Stat 
calculated the percentage of the iterations where the calculated difference was 
greater than the obtained difference. This is the probability that the obtained 
difference is a random occurrence. 
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Identifies the variables and iterations you selected. 

Shows the obtained means, sample sizes, differences between means, and 
probabilities for the groups identified by the grouping variable. Referring 
to the example above: 

Obtained means and sample sizes for each group are shown along the 
diagonal. The actual mean Effect Sizes for the Long, Normal, and 
Short groups are 1.4250, 0.4953, and 0.3540, respectively. 

The sample sizes for the Long, Normal, and Short groups are 7, 1 6, 
and 5. 



The difference between the obtained mean Effect Size for the Normal 
group (0.4953) and the obtained mean Effect Size for the Short group 
(0.3540) is (0.1413). In the Randomization analysis, Meta-Stat 
calculated that, over 500 iterations, there is a probability of 
approximately 65 percent that you would obtain this large a difference 
in Effect Size with randomly drawn data. In other words, the observed 
difference in Effect Size between these two groups is not unusual; there 
is a high probability of seeing this difference using randomly 
partitioned data. 

By contrast, the difference between the obtained mean Effect Size for 
the Normal group (0.4953) and the obtained mean Effect Size for the 
Long group (1.4250) is (0.9297). In the Randomization analysis, Meta- 
Stat calculated that, over 500 iterations, there is a probability of well 
under 1 percent (0.002) that one would obtain a difference this large 
with randomly partitioned data. In other words, the actual observed 
difference between these two groups is unusual, and one can assume 
that there is in fact some difference between these groups. 

NOTE: The approximate randomization test will tell whether the variables are 

independent. It is only appropriate for making inferences about a 
population when the data have been randomly selected. 
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Showing Correlations Between Variables 

Use a Correlation Analysis to investigate the relationship between variables. The 
analysis shows the degree to which variation in one variable is related to 
variation in another. 

Correlations can be positive or negative. A positive correlation means that a pair 
of variables tends to increase and decrease together. A negative correlation 
indicates an inverse relationship, in which one variable increases as another 
decreases. The size of the correlation measures the strength of the relationship. A 
correlation of 1 indicates a perfect linear relationship between the variables. 
Correlations that are close to 0 indicate little or no linear relationship. 

Correlation Analysis provides a good starting point for other analyses, including 
Regression. After using Correlation to discover strong relationships between 
variables, you can use Regression to calculate prediction equations for the 
variables. 

To perform a Correlation Analysis, follow these steps: 

1 . Select Analysis/Correlation. 

Meta-Stat displays a screen where you can identify the variables you want 
to use. 

2. Use Continuous Vars to identify continuous or equation variables for the 
analysis. 

Use Dummy Coded Vars to identify blocking or discrete variables for the 
analysis. Meta-Stat will treat each category within the variable as a 
separate dummy variable for the analysis. For more information, see the 
section "Using Dummy Variables" earlier in this chapter. 

3. If necessary, identify a weighting variable, a method to handle missing 
data, and a SELECT IF statement. For more information , see "Common 
Functions " earlier in this chapter. 
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Press F10 to display the results. 



META-STAT 



Listuise deletion 



09/10/93 



CORRELATIONAL ANALVSIS 



Correlation Matrix and Sample size CM) 





UNBIASED 


TOTAL J1 


HOURS 


UNBIASED 


1.0080 


0.8956 


0.7331 




C27) 


C27) 


(27) 


TOTAL Ji 




1.0000 


0.6604 






(27) 


(27) 


HOURS 






1.0000 








(27) 



Ctrl-P Print F10 Menu 



22:43:54 



u The screen shows the correlation coefficient between each pair of 
variables. In the example above, the correlation between Hours and 
Unbiased Effect Size is relatively strong (0.5406), while the correlation 
between T0TAL_N (Total Sample Size) and Unbiased Effect Size is weak 
(0.2423) . Larger numbers indicate a stronger correlation, with 1 meaning 
a perfect correlation and 0 meaning no linear correlation. 

The number in parentheses identifies how many studies were used for the 
sample. This number is determined by the number of studies in your meta- 
analysis and the method you used to handle missing data. For more 
information, see "Common Functions" earlier in this chapter. 

HINT : The correlation coefficient measures only linear (i.e., straight line) 

relationship between two variables. It is possible, though unlikely, that 
two variables can be perfectly curvilinearly related and yet their 
correlation equals 0. 
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To detect curvilinear relationships, you should inspect the Scatterplot 
option (see Chapter 8 ). When variables are curvilinearly related, you 
may wish to consider a transformation to "linearize*' them before 
performing multiple regression analyses. 



Cross-Tabulating Groups 

Use a Cross-Tabulation analysis to investigate the relationship between two 
discrete or blocking variables. 

The analysis displays a two-way table that shows the joint frequency 
distribution. You can use the table to review the distribution of values for the 
two groups. The analysis also provides a measure of statistical significance, to 
help you determine if the variables are statistically independent. 

To perform a cross-tabulation, follow these steps: 

1 . Select AnalysIs/CrossTabs 

Meta-Stat displays a screen where you can identify the variables you want 
to use. 

2. Use Grouping Variable 1 to identify the discrete or blocking variable that 
you want to use as the column variable in the two-way table. 

3 . Use Grouping Variable 2 to identify the discrete or blocking variable that 
you want to use as the row variable in the table. 

4. Optionally, identify a weighting variable and construct a SELECT iF 
statement. For more information , see "Common Functions" earlier in this 
chapter . 
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In the following example, a cross-tabulation will be performed using 
grouping variables for Publication Type and Hours: 



Uariable Analysis Charts Options Printer 




S 1111 ! j|||;||| ||| j j | Il| ifi® '-''"I 1 ;'; i«§ | f 

V^Fl-Help FlB-ttenu COACH 15: 44; 19 J 



4. Press F10 to display the results. 
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Results Screen 



f 1ETA-STAT 09/10/93 \ 

| CROSS TABULAT IOMS 1 

PUBTVJ3P Publication Type Ualid Observations : 30 Q 

by H0UR_6RP Duration of program 

Observed Cand Expected) Frequencies 
Publication Type 





O 


1 


2 


3 


5 


Row 


Duration of 


Missing 


journa 1 


dissert 


paper 


other 


Total 


0 Missing 


e 


0 


1 


1 


0 


2.00 




C0.07) 


(1.00) 


(0.40) 


(0.40) 


(0.13) 




1 short 


A 0 


2 


1 


2 


0 


5.08 




© ce.i7) 


(2.50) 


(1.00) 


(1.00) 


(0.33) 




2 auerage 


1 


11 


2 


1 


0 


15.00 




C0.50) 


(7.50) 


(3.00) 


(3.00) 


(1.00) 




3 long 


6 


2 


2 


2 


2 


8.00 




(0 .27) 


(4.00) 


(1.60) 


(1.66) 


(0.53) 




Column Total 


1.00 


m 


6.00 


6.00 


2.00 


30.00 


Ctrl-P Print 


F10 Menu U 


new grp 1 


«-* new grp 


2 




22:44:28 



O Identifies the variables you selected for the analysis, along with the 
number of studies for which data were available. 

© Shows the tabulated results, with row and column totals. The numbers in 
parentheses show the values that would be expected, given the row and 
column totals, if there were no relationship between the variables. Large 
differences between the observed and expected values may indicate a 
systematic relationship between the variables. 



Cross-Tabulating for Other Groups 

From the results screen, you can quickly flip through cross-tabulations for other 
groups in your meta-analysis. Press 1 or 1 to select a new column variable. Press 
the or cursor key to select a new row variable. 
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Calculating Group Means 

Use a Group Means analysis to calculate means for members of categories, or 
groups, that are identified by a discrete or blocking variable. The analysis 
provides you with the mean, standard deviation, and 95 percent confidence 
intervals. If you use Unbiased Effect Size as the criterion variable, the Group 
Means analysis also provides homogeneity measures. 

To use the Group Means analysis, follow these steps: 

1 . Select Analysis/ group Means 

Meta-Stat displays a screen where you can identify the variables you want 
to use. 

2. Use Criterion Variable to identify the variable that you want to measure for 
each group. Typically, this is the Effect Size. 

3. Use Grouping Variable to identify the discrete or blocking variable for 
which you want to calculate the means. 

Leave this field blank to calculate means for all discrete and blocking 
variables. 

4. If you left Grouping Variable blank, use the Sort Data field to identify a 
sorting sequence for the group members that Meta-Stat will display. 

5. Optionally, identify a weighting variable and construct a SELECT IF 
statement. For more information, see " Common Functions " earlier in this 
chapter. 
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In the following example, Meta-Stat will calculate the mean Effect Size for 
the TEST_GP variable: 



i » I .i«iuniiit|i.i,iMii l i|i HI .1 i 1 j J i j . i n u I . i i I I I i 1 i * I I I 1. ill i n < m i « t i 1 ! i;i mm i in mi i'm'i. m . i ii i oVi n i t i’i i i i i i -i I‘I t i . i i i i m i m in 
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, , I jsjjjj ■ 11 “ i * ' ijijll .... 
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ir — * — — - — Group Means .. — — { f msb : 

•ill v i'i'.i’.! i ‘I'iWh *!>' !>■!.[*’ .A. i . . 

Dependent Uariable: UNBIASED C]V£££££ 25 i£££ 2£££1 iii- iji.i 1 1 it, r. i mi | ii 1 1 1 i (' 



Dependent Uariable : 
Group ing Mar table : 

Sort Data : . Names 

Select If : 



UNBIASED 

TEST_GP 



Ascend ing 



>ts of 



. . ■ ' ; 
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6. Press F10 to display the results. 
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Results Screen 



f META-STAT 03/10/93 

GTOUP MEANS 

Criterion Variable: UNBIASED _ 

O 











Mean 


St. Deu. 


M 


952 Confid 


1 

i 

•— a 


TEST_GP Test constructor 
















1 


teacher 


0.793 


0.633 


6 


0.161 


to 


1.425 




2 


standard I 


0.672 


0.702 


11 


0.206 


to 


1.138 


© 


3 


other 


0.416 


0.547 


12 


0.071 


to 


0.760 






© 




















1 Hedges's Honogenity Tests - 
















Source 


Statistic 


df 
















Total 


116.965 


28 Cp = 0.000) 














Uithin 


107.799 


26 Cp = 0.000) 














Between 


9.166 


2 Cp = 0.010) 









Ctrl-P Print FID Hern 



22:46:34 









O 



This screen identifies the criterion variable you are measuring for each 
group member. 
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© 



For each group member, Meta-Stat displays the following information: 



Value 


Description 


Mean 


Mean value, equal to the sum of the 
values divided by the number of cases 
(studies). The mean is the most 
common measure of central tendency. 


Stand. Dev. 


Standard deviation, equal to the 
positive square root of the variance. 
(The variance is a measure of the 
dispersion of values around the mean, 
roughly equal to the average squared 
deviation from the mean.) The smaller 
the standard deviation, the more 
homogeneous the data. 


N 


Number of group members 


95% Confid Int. 


The 95 percent confidence intervals for 
the group means. 



© Hedges's Homogeneity Test measures the total variation among the groups, 
as well as the variation within and between the groups, to test the 
hypothesis that the Effect Sizes come from the same population — that is, 
that they are homogeneous. If the statistic is large ( or equivalently if the 
value of p is low, say less than .05), Hedges's test indicates that there is a 
low probability that the Effect Sizes are from the same population. 

Hedges and Olkin note (1985, pages 157-159) that the homogeneity 
statistics can be used for fitting models to effect size for groups in a 
procedure that is analogous to the procedure used to fit hierarchical log- 
linear models to contingency tables. They outline a strategy similar to the 
following: 

Step 1 : Look at the total homogeneity statistic (Q T ). If the value is 

not large, i.e. is not statistically significant, then you are 
done. You can conclude that the model of a single Effect Size 
fits the data. 

Step 2: If Q t is large, then look at the within homogeneity statistic, 

Qw If Q w is not statistically significant, then you can you 
can conclude that the model of a different Effect Size for 
each of your groups adequately fits the data. The between 
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fit statistic, Q b , indicates the extent to which the effect sizes 
differ among groups. 

A large value of Q w indicates that effect sizes are not homogeneous within 
the groups. You should redefine your groups, either through secondary 
classifications, using a different grouping variable, creating new groups, or 
using the SELECT IF feature. Then again, compute Qw 



Results Screen for All Grouping Variables 

If you did not specify a grouping variable, Meta-Stat shows results for all 
discrete and blocking variables. Use PgDn to move from one screen to the next: 

NOTE: If you are interested in analyzing the UNBIASED Effect Size as the 

criterion variable, Meta-Stat will compute Hedges’s homogeneity 
statistics. You do not need to specify WEIGHT as a weighting variable 
as this is done internally to perform the necessary calculations. 



Calculating Frequencies 

Use Frequency analysis to count the number of different values for a variable 
and display the frequency at which each value occurs. This analysis gives you a 
clear picture of the breakdown of data for a variable. 

Follow these steps: 

1 . Select Analysis/ Frequencies 

Meta-Stat displays a screen where you can identify the variable you want 
to count. 

2. Use Criterion Variable to identify the variable that you want to count. This 
can be any type of variable. 

3 . If necessary, identify a weighting variable and construct a SELECT IF 
statement. For more information, see "Common Functions " earlier in this 
chapter . 
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In the following example, Meta-Stat will count PUBTY_GP: 
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Press F10 to display the results. 



META-STAT 



Criterion Variable: 



Value Label 

Missing 
1 Journal 
gt 2 dissert 
EJ 3 paper 
5 other 



09/10/93 



FREQUENCIES 

PUBTYJGF Publication Type 

O 

Sun of Relative 

Jases Weights Frequency 

1 1.008 3.3* 

15 15.000 50.0* 

6 6.000 20 . 0 * 

6 6.000 20 . 0 * 

2 2.000 6.7* 



N of Cases: 30 



Adjusted 

Frequency 

51.7* 

20.7* 

20.7* 

6.9* 



Ctrl-P Print F10 Menu 



22:47:10 



The results screen: 



O Identifies the variable you are measuring. 

© Displays the following information for each value of the variable: 
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Value 


Description 


Value 


A value for the variable. 


Label 


For discrete and blocking variables only, 
shows the label attached to the value. 


Sum of 


The sum of the weighting data that you 


Weights 


used, if any. 


Relative 


The relative frequency with which the value 


Frequency 


occurs. 


Adjusted 


The relative frequency with which the value 


Frequency 


occurs, adjusted after removing missing 
values. 



Displaying Means lor a 
Two-Way Breakdown of Groups 

The Breakdown analysis is similar to the Cross-Tabulation analysis in that it 
provides a two-way breakdown of grouping variables. However, the Breakdown 
analysis provides more detail on the data by calculating, for each two-way 
breakdown, the mean for the criterion variable. For example, where Cross- 
Tabulation provides you with the number of cases for each two-way breakdown, 
the Breakdown analysis calculates means for the criterion variable for each two- 
way breakdown. 

To perform the analysis, follow these steps: 

1 . Select Analysis/Breakdown 

Meta-Stat displays a screen where you can identify the variables you want 
to use. 

2. Use Criterion Variable to identify the variable you want to count. 

3. Use Grouping Variable 1 to identify the discrete or blocking variable that 
you want to use as the column variable in the two-way table. 
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4. Use Grouping Variable 2 to identify the discrete or blocking variable that 
you want to use as the row variable. 

5 . Optionally, identify a weighting variable and constructing a SELECT IF 
statement. For more information, see " Common Functions ” earlier in this 
chapter . 

In the following example, Meta-Stat will calculate the mean Effect Size for 
each cell in the two-way breakdown of H0UR_GRP and TEST_GP: 
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Results Screen 



f META-STAT 09/10/33 ^ 

BREAKDOWN 

UNBIASED Unbiased Effect Size Ualid Observations: 29 
by HOUR_GRP Duration of program A 
TESTJGF Test constructor w 

Means Cand Cell Sizes) 







Duration of 


program 








Q 


1 


2 


3 


Bow 


Test constr 


Missing 


short 


average 


long 


Total 


1 teacher 


0.909 




0.325 


1.436 


0.793 




Cl) 




C3) 


C2) 


C6) 


2 standard 


0.192 


© 


0.462 


1.322 


0.672 




Cl) 




C7) 


C3) 


Cll) 


3 other 


. 


0.178 


0.428 


0.980 


0.416 






C5) 


C5) 


C2) 


02) 


Column Total 


0.550 


0.178 


0.423 


1.257 


0.591 




C2) 


C5) 


C15) 


C7) 


C29) 



Ctrl-P Print F10 Menu U new grp 1 ++ new grp 2 



22:47:44 









The results screen: 

O Identifies the variables you selected for the analysis, along with the 
number of studies for which data were available. 

© Shows the tabulated results for each breakdown, along with means for 
each row and column. The numbers in parentheses show the cell sizes. 
For example, in studies where teachers devised tests, the mean Effect Size 
was 1 .108. However, where the duration of the program was long, the 
mean Effect Size was 1.958. Long durations, in general, were associated 
with higher Effect Sizes (a mean of 1,475). 
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Showing Breakdowns for Other Groups 

From the results screen, you can quickly flip through breakdowns for other 
groups in your meta-analysis. Press t or 1 to select a new column variable. Press 
-*■ or «- to select a new row variable. 
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Charting 

Data 



Meta-Stat offers a variety of powerful charting features to help you explore — and 
present — your data. 



Before You Begin 

Read the section "Common Functions " at the beginning of Chapter 7. That 
section describes common Meta-Stat functions that apply to statistical analyses, 
which are covered in Chapter 7, as well as to charts, which are the subject of this 
chapter. In particular, you should know how to: 

• Select variables to be analyzed and charted. 

• Deselect variables that are already selected. 

• Weight data using a weighting variable. 

• Select data with SELECT IF statements. 
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Setting Chart Options 

Before creating charts, you can change the Meta-Stat chart options. 



Video Mode 

Meta-Stat can display charts in a variety of video modes. Select a mode that is 
compatible with your display adapter. 

NOTE: When you save a chart by pressing Ctrl-S, Meta-Stat saves the chart in 
VGA resolution (640 x 480), regardless of the video mode you are 
currently using. 

Follow these steps to set the video mode: 

1 . Select Charts/VKdeo Modes 

Meta-Stat displays a list of modes you can use: 
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2. Highlight the mode you want to use, then press Enter. 



PCX Colors and Size 

When you save a chart by pressing Ctrl-S, Meta-Stat saves the chart as a PCX- 
formatted file. You can then import this file into other applications, or view it 
later from within Meta-Stat. Meta-Stat can save the chart in color or in black 
and white. Typically, you would use color if you wanted to produce color slides, 
and black and white if you wanted to reproduce charts for a published article. 

You can also specify the size of your charts on printed output. 

Follow these steps: 

1 . Select Charts/ PCX Colors 

2. Highlight Black & White or Color, then press Enter. 

3. Select Charts/Graph Size 

4. Specify the height and width you want to use, then press Enter. 



Point and Shoot Resolution 

When you create a scatterplot, you can display the study data for a particular 
data point. You do this by using your mouse to "point and shoot" at the data 
point. 

You can set the point and shoot resolution so that your mouse points at different- 
sized areas. By setting a wide resolution, for example, you can select many data 
points at once — even data points that are not directly below the mouse pointer. 
With a narrow resolution, you must move the mouse pointer directly to a data 
point before you can select it. 

NOTE: Meta-Stat displays a mouse pointer only if you create the scatter chart 
when in CGA, VGA, or Hercules video mode. You do not see the pointer 
when you use either of the two Super VGA (SVGA) modes. 
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Follow these steps to select the point and shoot resolution: 

1 . Select Charts/P&S Resolution. 

2. On the screen that Meta-Stat displays, type a number from 1 to 10, then 
press Enter. 

Larger numbers provide wider resolution. Smaller numbers provide more 
narrow resolution. 



Saving Charts 

After you create and display a chart, you can save it by pressing Ctrl-S. Meta-Stat 
saves the chart in a PCX-formatted file. You can then import this file into 
another application, or you can view it later with Meta-Stat . 

Meta-Stat saves the PCX file in the Meta-Stat directory. It selects the filename 
for you: 

• The filename extension is PCX. 

• The beginning of the name identifies the type of chart — BAR for a Bar chart, 
BOOT for a Bootstrap chart, LINE for a Line plot, and so on. 

• A number identifies the file uniquely. For example, BAR0.PCX is the first 
Bar chart you created, BARI .PCX is the second, and so on. Meta-Stat 
selects this number for you, based on the charts you have already saved. 
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Viewing and Deleting Charts 

After you save charts, you can view them from within Meta-Stat. You can also 
delete them if you no longer need them. 



Viewing a Saved Chart 

1 . Select Charts/View Saved Chart 

Meta-Stat displays a list of charts that you have saved: 
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2. Highlight the name of the PCX file that you want to view, then press Enter. 

NOTE: Meta-Stat saves charts in VGA resolution (640 x 480). For best results, 
switch to VGA resolution before viewing a chart. If you view a saved 
chart while using a different resolution, the chart will be either smaller 
or larger than your screen. 
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Deleting a Saved Chart 

1 . Select Charts/Delete Saved Chart 

Meta-Stat displays a list of charts that you have saved: 
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2. Highlight the name of the PCX file that you want to delete, then press Enter. 

3. Meta-Stat asks you to confirm (Y or N). 
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Creating a Bar Chart 
(Frequency Analysis) 

Meta-Stafs Bar chart can display the frequency with which values occur for a 
variable. Follow these steps to create a Bar chart: 

1 . Select Charts/ Bar (freq). 

Meta-Stat displays a screen where you select the variable you want to chart. 

2. Use Criterion Variable to identify the variable you want to chart. 

3 . If necessary, identify a weighting variable and construct a SELECT IF 
statement. For more information, see ''Common Functions" at the beginning 
of Chapter 7. 

In the following example, Meta-Stat will chart the Unbiased Effect Size: 
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4. Press F10 to display the results. 

Notice that Meta-Stat aggregates the values of the variable and selects an 
appropriate scale for the X axis: 



r 







5. Press 1 to select and chart a new variable. 
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Creating a Bootstrap Chart 

To create the Bootstrap chart, Meta-Stat uses a brute-force resampling technique 
to calculate expected effect sizes for the population of studies in your meta- 
analysis. For up to 1,000 iterations, Meta-Stat randomly selects an Effect Size 
from your population of studies, then records the frequency at which each Effect 
Size is picked. 

Follow these steps to create the chart: 

1. Select Charts/ Bootstrap. 

Meta-Stat displays a screen where you select the variable you want to chart. 

2. Use Criterion Variable to identify the Effect Size variable. 

3. If necessary, identify a weighting variable and construct a SELECT IF 
statement. For more information, see " Common Functions " at the beginning 
of Chapter 7. 
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4. Press F10 to display the chart. 

In the following example, 100 iterations were used. Notice the tallest bar on 
the chart. This bar indicates that, of the 100 picks, the Effect Size 0.75 was 
picked 7 times. 




NOTE: The bootstrap procedure assumes the data were randomly selected. 
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Charting Effect Sizes by Study 

Use the Effect Size by Study chart to plot the Effect Size and 95% confidence 
intervals for the mean of the studies in your meta-analysis. Follow these steps: 

1 . Select Charts/ Effect Size by Study. 

2. Use the Sort Data field to specify a sorting sequence for the chart. 

You cannot specify a different criterion variable. The Effect Size variable is 
always used. 

3. Press F10 to display the results. In the following example, effect sizes are 
sorted by value, in descending order: 



r 



Effect Size and 95'/ Confidence Interval 




EMERY “ 
TH0MAS3- 
J0NGS2- 
SCOTTft “ 
THOMAS 1 - 
BAJTEL2- 
MCPHAIL1- 
SLAUGH- 
CALLEN- 
BAJTEL3- 
JOHNS- 
-Q LAGANA- 
• rH JONGS 1- 

RAYFORD” 
BAJTEL1 - 
"O DILLAR — 

3 LEUINE- 
■*-' MCPHAIL2- 
W GARLT- 
SPIEL- 
ALLEN- 
COSTAR- 
EAKIN- 
JACOBS - 
GROSS - 
ROMBERG “ 
MANN- 



+ 



+ 

+ 

+ 






H 

H 1 

1 1 

1 — 

h 



i — I — i 
. 1 



CR02IERH 
TH0MAS2 H 



+ 



| i i i i i i i i i I < i i i i i i i i | i i i i i i i i i | i i i i i rn i | i i i i i ir i i | m i i rrn'rj'T i r 
-0.9 -0.1 0.1 0.6 1.1 1.6 2.1 






Effect Size 






Chapter 8 



BEST COPY AVAILABLE 
205 



193 




Charting Data 



Meta-Analysis User's Guide 



Showing Relationships on a Line Graph 

Use a Line graph to show the relationship between an predictor variable and the 
criterion variable, Effect Size. You can use discrete and blocking variables to 
group data into separate plots on the same chart. 

1 . Select Charts/ LineGraph. 

Meta-Stat displays a screen where you identify the variables and other 
information for the chart. 

2. Use the Criterion (y) Variable field to specify the variable you want to plot on 
the Y axis of the chart. Typically, this is the Effect Size. 

Use Transform y to apply various transformations to this variable. You can 
also smooth the results with Smooth y. 

3 . Use the Predictor (x) Variable field to specify the variable you want to plot on 
the X axis of the chart. 

Use Transform x and Smooth x to transform and smooth the variable. 

4. Optionally, use the Grouping Variable field to identify a discrete or blocking 
variable. Meta-Stat will separately plot each category within the variable. 
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5. If necessary, use SELECT IF to select data for the chart. 

In the following example, Meta-Stat will chart the relationship between 
EFFECTSZ (Effect Size) and EXP_N (the size of the experimental group): 




6. Press F10 to display the results. 
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Results for X and Y Variables 

If you specified only an X and Y variable, Meta-Stat shows a single plot: 
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Results with a Grouping Variable 



If you specified a grouping variable in addition to X and Y variables, Meta-Stat 
shows one plot for each category identified by the grouping variable. For the 
following example, a blocking variable called Grade Group was specified. This 
variable identifies three groups of subjects: elementary, high school, and college 
subjects. 
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Selecting Other Variables 

Once you have displayed the chart, you can quickly select other variables for the 
chart: 



Press this key 


To select a new ... 


1 or 1 


Criterion (Y) variable 


-* or «- 


Predictor (X) variable 


space or backspace 


Grouping variable 



NOTE: To look up the data behind a data point, plot the same variables on a 
scatterplot. Then "point and shoot" with your mouse on the data point. 

When multiple values of the criterion variable have the same value on 
the predictor variable, Meta-Stat plots their average. It is good practice 
to examine the Scatterplot of the same data. 



Plotting Data Points on a Scatter Plot 

Use a Scatter plot to plot individual data points that show the relationship 
between a predictor variable and the criterion variable, typically Effect Size. After 
you display the chart, you can use your mouse to display study data by "pointing 
and shooting" at a data point. 

1 . Select Charts/Scatter Plot. 

Meta-Stat displays a screen where you identify the variables and other 
information for the chart. 

2. Use the Criterion (y) Variable field to specify the variable you want to plot on 
the Y axis of the chart. Typically, this is the Effect Size. 

Use Transform y to apply various transformations to this variable. 

3. Use the Predictor (x) Variable field to specify the variable you want to plot on 
the X axis of the chart. 

Use Transform x to transform the variable. 
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4. If necessary, use SELECT IF to select data for the chart. 

In the following example, Meta-Stat will create a scatter plot showing the 
relationship between EFFECTSZ (Effect Size) and EXP_N (the size of the 
experimental group): 




5 . Press F10 to display the results: 
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Selecting Other Variables 

Once you have displayed the chart, you can quickly select other variables for the 
chart by pressing t 1 or . 



Pointing and Shooting at Data Points 

If you create a Scatterplot while in CGA, VGA, or Hercules video mode, you can 
move the pointer — shown as a magnifying glass — to a data point and left-click. 
Meta-Stat displays the study (or studies) represented by the data point. 

If the magnifying glass was pointing at more than one data point, Meta-Stat 
displays the study for the first data point. Press F8 to display additional studies. 
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Showing the Median, Range, 
and Quartiles with Box & Whiskers 

Use the Box & Whiskers chart to display the median, range of values, and 
percentiles for a variable. You can also use discrete and blocking variables to 
group data into separate Box & Whiskers on the same chart. 

To create the Box & Whiskers chart, follow these steps: 

1 . Select Charts/Box & Whiskers. 

Meta-Stat displays a screen where you identify the variables and other 
information for the chart. 

2. Use the Criterion Variable field to specify the variable you want to measure. 

3. Optionally, use the Grouping Variable field to identify a discrete or blocking 
variable. Meta-Stat will create a separate graphic for each category within 
the variable. 

4. If necessary, use Select If to select data for the chart. 

In the following example, Meta-Stat will measure the unbiased effect size: 
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In the following example, Meta-Stat will measure the unbiased effect size: 
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5. Press F10 to display the results. The results are displayed as follows: 



Graphic Feature 


Description 


Top of Whisker 


The maximum value 


Bottom of Whisker 


The minimum value 


Top of Box 


The 75th percentile, the third 
quartile 


Bottom of Box 


The 25th percentile, the first 
quartile 


Line inside the Box 


The 50th percentile, the median 
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Results for a Criterion Variable 

If you specified only a criterion variable, Meta-Stat shows a single Box & 

Whiskers: 
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Results with a Grouping Variable 

If you specified a grouping variable in addition to the criterion variable, Meta- 
Stat shows a separate graphic for each category identified by the grouping 
variable. For the following example, a grouping variable called Hour Group was 
specified. This variable identifies the duration of the intervention program: 
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Selecting Other Variables 

Once you have displayed the chart, you can quickly select other variables for the 
chart: 



Press this key 


To select a new ... 


t or i 


Criterion variable 


space or backspace 


Grouping variable 



Displaying the Mean Values for Groups 

Use the Means chart to show the mean value for one or more groups, as are 
represented by discrete or blocking variables. Meta-Stat measures the value for 
each group. The chart also shows the 95% confidence interval. 

To create the Means chart, follow these steps: 

1 . Select Charts/ Means. 

Meta-Stat displays a screen where you identify the variables and other 
information for the chart. 

2. Use the Criterion Variable field to specify the variable you want to measure. 
Use Transform y to apply various transformations to this variable. 

3. Optionally, use the Grouping Variable field to specify a discrete or blocking 
variable. Meta-Stat will show the means only for the groups identified by 
this variable. 

If you leave this field blank, Meta-Stat will show the means for all discrete 
and grouping variables. 

4. You can weight, sort, and select data for the chart, if necessary. 
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In the following example, Meta-Stat will show the mean effect sizes for a 
variable called TEST_GP: 
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5. Press F10 to display the result. 
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Results for a Single Grouping Variable 

If you specified a grouping variable, Meta-Stat shows the means for categories 
within that group: 
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Results with a Grouping Variable 

If you did not specify a grouping variable, Meta-Stat shows the means for all 
categories within all groups. In the following example, the data are sorted by 
value, in descending order: 
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Selecting Other Variables 

Once you have displayed the chart, you can quickly select other variables for the 
chart: 



Press this key 


To select a new ... 


T or I 


Criterion variable 


space or backspace 


Grouping variable 
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This chapter outlines various equations and procedures used in Meta-Stat . The 
equations are based on generally accepted statistical principles and on methods 
described in Glass, McGaw, and Smith (1981), Hunter and Schmidt (1990), Cohen 
(1977), Veldman (1967), Hedges and Olkin (1985), Draper and Smith (1966), and 
SPSS (1991). The reader is referred to these excellent materials for a fuller 
development of the technical approaches used in Meta-Stat . 



Conversion Formulas 

Meta-analysis is based on placing the results from different research studies on a 
common metric. Meta-Stat allows the user to use either Effect Size or correlation 
as that common metric. To calculate Effect Size from a t, F, or p-value , the 
reported statistics are first converted to a correlation (r) and the correlation is 
converted to an Effect Size (d). Taylor and White (1993) found that the 
alternative methods for computing Effect Size yield very comparable results. 

The following formula is used to convert from an r to Effect Size: 



d= 



4 r 



N 1 -r 



Correlational meta-analysis simply uses the r as the common metric. 
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• To derive r from a f-statistic: 



r 






N 



t 2 /(.l-r v f + df 



• To derive r from an ^-statistic: 



r 



\ F + df 



Here df refers to the denominator degrees of freedom. The equation is only 
appropriate when there is 1 degree of freedom in the numerator. 



• To derive r from a ^-statistic with 1 degree of freedom: 

r = JtfiN 



• To derive r from a probability level (p), p is first converted to a z and the z is 
then converted to an r: 



= fl _ 2.515517 + .802853a+. 010328a 2 

1 + 1 .432788a + . 1 89269a 2 + ,001308a 3 

where 



a = \j In (1 / p 2 ) 

r = f?l N 



o 



9 0 
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• To compute Effect Size as the standardized mean difference: 




°c -5 

• To compute Effect Size as parametric gain scores: 

X — y x — y 

£ _ ej>ost ej>re _ cj>ost cj>re 

a a 

ejyre cj>re 



• To compute Effect Size from a paired comparison t-test 

d-bH 

t 

where ^ is the mean difference between the observed pairs. 
This last equation is from Gibbons, Hedeker, and Davis (1993). 



Pre-Coded Variables 

There are precoded equations in both the Effect Size and correlational meta- 
analysis modules. Here, we discuss these equations and outline their use. 



Effect Size Meta-Analysis 

Two pre-coded variables are used in the Effect Size meta-analysis module, 
UNBIASED, and WEIGHT. Both of these variables are based on the work of Hedges 
and Olkin (1985). 

The Unbiased Effect Size, d \ is used in several homogeneity calculations and as 
the bases for the Effect Size plot. It should be noted that d ''usually differs only 
slightly from d, especially as N gets large (see Hedges and Olkin, 1985, pp 78- 
79.). Taylor and White (1993) found that d 7 and d produce the same conclusions. 
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d 1 = d 

where N, is the total sample size 



- 9 J 

and d is the original effect-size. 



The variable WEIGHT is the inverse of the variance. The variance is used in the 
Effect Size plot. The optimal weight when analyzing d 'and estimating the true 
Effect Size is the inverse of the variance. The most precise studies are given the 
greatest weight. The more precise studies are give higher weights (see Hedges 
and Olkin, 1985, pp 302-304). 



WEIGHT = 



2 JV, 2 + N/J c (df 



Correlational Meta-Analysis 

The correlational meta-analysis modules have numerous precoded variables, 
including adjusted correlation, weights, and attenuation formulas. 



Adjusted Correlation and Weights 

The adjusted correlation is the unadjusted correlation divided by the product of 
the correction factors. This is the what the correlation would be if it were not for 
artifacts in the data. 

Two suggested weights are provided based on discussions by Hunter and 
Schmidt (1990, pages 145-150). WEIGHTB is the optimal weight presented by 
Hedges and Olkin (1985): 



W L 



. N t - 1 
CF 2 - ' 



2\2 



(1 ~ r ) 



CF, the product of the correction factors (attenuation factors to use the 
terminology of Hunter and Schmidt), is discussed in the next section. 

In the case of multiple independent variables, W b is multiplied by 



1 + r 2 +(-— -1) 



V 
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Hunter and Schmidt show that WEIGHTA = N t CF 2 can differ only trivially from 
WEIGHTB and has the desired effect -- the greater the correction needed for a 
study, the less weight it is given. As with the Effect Size weight above, you may 
want to use one of these weights to obtain an improved true value estimate of the 
corrected (adjusted) correlation. 



Attenuation Formulas 

Attenuation formulas are used in correlational meta-analysis to adjust for 
characteristics of the data (artifacts to use the terminology of Hunter and 
Schmidt). They provide a theoretical estimate of the attenuation in the true 
correlation due to data characteristics. 

If r Q is the observed correlation,/? is the true correlation, and CF is a correction 
factor for some data characteristic, then p - r/CF. Correction factors can be 
computed for multiple data characteristics and are assumed to be independent. 

• To correct for unreliability in either the dependent or independent variable: 

CF-fZ 



where is the reliability of the measure used to assess the dependent or 

independent variable. 

• To correct for variable dichotomization (i.e., to assess the correlation for a 
continuous variable given a point biserial correlation): 

CF = 

y/P(PP) 

where c is the point in the normal distribution that divides the 

distribution into proportions P and 1-P, i.e., the inverse of the 
cdf (values for c are derived using a look-up table), and 

-±c 2 

<1 >(C) = ~^=~ 
yj2lt 

i.e., the normal ordinate at c. 
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• To correct for the use of a proxy dependent or independent variable: 




where is the correlation of the proxy variable with the variable it is 
replacing and is the reliability of the proxy variable. 

• To impartial a partial correlation: 

cf = i / 

where is the correlation of the extraneous variable with the dependent 
variable. 

• To correct for restriction in range for the independent variable: 

CF = y' u 2 + r 0 2 (l - u 1 ) 

where u is the ratio of the study standard deviation to the 
referencestandard deviation. 



Probability 

Probabilities for various statistics are based on Kendall's (1955) normalizing 
transformation of the F distribution: 



z 



fl- 


—)f 113 - 1 




\ mi 1 


ft a) 


A 


2l. f ™ + 


2 


95 


9A 



where A is the df for the numerator of the F ratio, and B is the df for the 
denominator. 
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If B<4 then Z is transformed using Kelley's (1947) correction 



.08z 5 
= z+ 

B 3 



The z or z* then provides the basis for approximating the p-level using: 

p = .5/(1 + . 196854z+ .1151 94z 2 + .000344z 3 + .01 9527z 4 ) 4 

The procedure is used to provide the exact probability of other statistics: 

z Since F Jt00 = z 2 = tl, the probability for a z is computed using A=l, 
B=1000, F=z? . 

Since F = %/df with (df,«>) degrees of freedom, the probability for a chi- 
square statistic is computed using A=df B=1000 , F=tf/df. 

t Since F = t 2 with (l,df) degrees of freedom, the probability for a t- 
statistic is computed using A-l, B=df } F=F. 



Statistical Analyses 

The following identifies the equations used in the analysis module of Meta-Stat 



Descriptive Statistics 

This procedure provided the weighted and unweighted mean, median, variance, 
standard deviation, minimum value, maximum value, and range given a specified 
variable and an optional weighting variable. 

The computational formula used to derive the mean and variance are: 



Mean = (x { w ) 1 £ w i 



where is dependent variable data and is the weight for observation / 
(w; is set equal to 1 for all i if a weighting variable is not 
specified) 



r 2 _ 






r 
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The standard deviation is simply the positive square root of the variance. This is 
the standard deviation of the presented data. To treat this data as a sample and to 
obtain an unbiased estimate of the population variance, you can multiply the 
sample variance by n/(n-l ). 

If the dependent variable is the unbiased Effect Size then a variety of additional 
statistics are available. Hedges’s Q T can be used to test the homogeneity of the 
adjusted effect sizes. 



q t = E 



{d' - d'f 



where oj. is the variance of d 'and is computed as the inverse of WEIGHT. 
The computational formula used by Meta-Stat is 




E4 






E 1/o w/ 



If you page down after the presentation of the basic descriptive statistics for the 
adjusted Effect Size, another page of meta-analysis statistics appears. 



Mean ES or mean Effect Size is the mathematical average described above. It is 
repeated on this screen for reference. To compute the Fisher's Z, each Effect Size 
is first converted to an r using the reverse of the formula above: 



r 



d 

\/4 + d 2 



The r is then converted to an individual Z using 



Z = .5 In 



1+r 

l-r 



Fisher's Z is then computed as the weighted average of the individual Z’s. The 
probability test the null hypothesize of no population effect. By converting the 
Fisher's Z back to an Effect Size or an r (equivalent es and equivalent r) } we have 
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a refined estimate of the population Effect Size and correlation. The Failsafe N 
identifies the number of studies with opposite conclusions that would be needed 
to overturn a rejected null hypothesis. 



Fail safe N = 



N, (Z 2 - 3.8416) 
3.8416 



The variance ratio is the ratio of the Effect Size sampling error variance a 2 e to the 
variance of the overall mean Effect Size. The error variance is computed as 



o 



2 

e 



iitl 

,N-3 





where N is the average sample size, and d is the mean effect-size. If 
EFFECTSZ rather than UNBIASED is the criterion variance, d is corrected 
by dividing by 1 + . 75/(N-3). 



The sampling error variance is discussed by Hunter and Schmidt (1990, page 
281-338). 



Regression 

Meta-Stat uses the iterative stepwise multiple regression approach developed by 
Greenberger and Ward (1956) and described in Veldman (1967, pages 294-307). 
The iteration process begins by selecting the variable with the highest correlation 
available from the set of predictor variables. In subsequent iterations, Meta-Stat 
either selects the predictor variable which will maximally increase the square of 
the multiple correlation when added to the already selected predictor variables or 
adjusts the weight of one of the previously selected predictor variables. The 
process is terminated when neither adjustments nor the addition of variables 
increases the squared multiple correlation by more than .0000 1 . 

The key advantage of this iterative approach is that it allows the computation of 
key regression statistics (R 2 and beta weights) without having to invert the 
correlation matrix. Thus the non-multicolinearity restriction is not needed for 
computing the multiple correlation and accuracy is improved. The standard 
errors for the beta weights, and hence the corresponding t-statistics, however, are 
based on the inverted the correlation matrix. The Gauss-Jordon procedure is used 
to do the inverting. 
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Some relevant equations are: 

• Adjusted R-Square: 

Adjusted R 2 = R 2 — — 

N-p+\ 

where N is the number of observations and p is the number of included 
predictor variables. 



• Residual Sum of Squares: 



SS. = (l-R 2 )(C-l)S v 



where Syy is the sample variance for the criterion variable and C is the 
sum of the weights. 



• Regression Sum of Squares: 



• Mean squares 



SS R = R 2 (C-l)S t 



MS = SS/df 



• Standard error of the unstandardized regression weight: 






(N-P- 1) 



where aa^ is the kth diagonal element of the inverted correlation matrix 
and S k is the sample variance for the kth predictor variable.. 



• t-Statistic for testing H 0 : 6 k = 0 
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If the criterion variable is UNBIASED and the weighting variable is WEIGHT, then 
we follow the recommendations outlined on page 174 of Hedges and Olkin 
(1985). The weighted sum of squares about the regression is the chi-square 
statistic Q e and the weighted sum of squares due to the regression Q R . The 
corresponding probabilities are reported. The standard errors of the regression 
weights are corrected by dividing by the square root of the mean square error. 



Group Means 

This module provides means, standard deviations and confidence intervals by 
group for the selected criterion variable. Means and standard deviations are 
computed using the routines from the descriptive statistics discussed above. The 
95% confidence intervals are computed using 



c.i . 



X±t 



h-1,.05 




Values of t are obtained using a look-up table. 

If the criterion variable is the UNBIASED Effect Size, Meta-Stat also computes 
Hedges's within, between and total homogeneity statistics (Hedges and Olkin, 
1985, pp 153-165). The total homogeneity statistic tests whether the studies, 
regardless of the grouping variable, share the same Effect Size and is discussed 
above under descriptive statistics. 



Hedges's between homogeneity test is analogous to the analysis of variance F- 
test examining whether the group means are the same. The tested hypothesis is 
that the group means are the same. The test statistic is: 





where dj is the group mean weighted by the inverse of the variance, d is the 
grand mean weighted by the inverse of the variance, and 0 2 (dj,) is the 
variance of the unbiased effect size estimator. 



Q b is tested with p groups- 1 degrees of freedom. 
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Hedges within homogeneity test examines the hypothesis that the effect sizes are 
homogeneous within group. The test statistic is; 

i J O 

Q w is tested with N-p degrees of freedom. 

Hedges and Olkin (1985, p 156) note that if the samples sizes are at least 10 per 
group and if the effect sizes are not too large, then the actual significance of 
these test statistics will be sufficiently close to the nominal values that would be 
obtained from a large sample distribution. 



( 4 ,) 



Graphical Analysis 

Descriptions of the various graphs can be found in Chapter 8. The following 
outlines some of the equations used in the program. 



Effect Size Plots 

The approximate 95% confidence interval for the unbiased effect size is given by 

d ' - z anPd' < 6 < d ' + z an a d> 

where z an is the two-tailed critical value of the standard normal distribution 
and o d ' is the positive square root of the variance of d Computed as the 
square root of the inverse of WEIGHT. 



Mean Plots 

The approximate 95% confidence interval about the mean is given by 

* - Z an°n-l < V- < * + •WVl 

where z^ is the two-tailed critical value of the standard normal distribution 
and o n _, is the standard deviation estimate. 
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